anchorQueue.idx=0, next retry in...

[NoltaPulta]

2022.99.99-dev-bcb3cac2 and other devel branch recent commits.

Enters endless loop once it starts hitting these links/anchorQueue messages, rewinding blocks using integration doesn't fix the issue, tried state_stages, state_headers, etc.

At first, I was experiencing this issue syncing polygon from archive backup at https://snapshots.matic.today/. Then, when rewinding failed, tried syncing from blank slate w/snapshots, same issue, but occurs much earlier in the block history, e.g. INFO[08-26|14:14:53.381] { 1060514-1361695 links=301182 (1060514-1361695)} => 84c050e99f3d1476fe548370e7bff71ba29505dba559a6a08079266c349274ad, anchorQueue.idx=1, next retry in 4.999825896s INFO[08-26|14:14:53.381] {31397105-31834289 links=437185 (31397105-31834289)} => f21cda8b4db9afc6af33ead02185b2612fd49bca487bd022a0ae5b261782f924, anchorQueue.idx=0, next retry in 4.406113172s

Have been trying to get polygon working for the past week or so, encountering at the above and similar errors.

Then, today, same issue happened while syncing eth mainnet, which had not ever been a problem until now.

[AskAlexSharov]

not enough logs

[NoltaPulta]

INFO[08-27|12:53:05.899] [1/16 Headers] Waiting for headers... from=32034415 INFO[08-27|12:53:32.845] [txpool] stat block=32034415 pending=0 baseFee=0 queued=0 alloc=3.7GB sys=4.7GB INFO[08-27|12:54:29.334] [p2p] GoodPeers eth66=4 INFO[08-27|12:54:32.845] [txpool] stat block=32034415 pending=6 baseFee=2274 queued=2078 alloc=4.4GB sys=4.7GB INFO[08-27|12:54:45.900] Req/resp stats req=551 reqMin=32035182 reqMax=32388135 skel=15 skelMin=32035951 skelMax=32330671 resp=1071 respMin=32089327 respMax=32135790 dups=0 INFO[08-27|12:54:45.903] Queue sizes anchors=139 links=64651 persisted=65536 INFO[08-27|12:54:45.903] {32035375-32035951 links=577 (32035375-32035951)} => f8edfb155e33221b88fb364770c01ab0217eb83f7df6136a86ba7cfe37c57cce, anchorQueue.idx=0, next retry in 1.660512011s INFO[08-27|12:54:45.903] {32038447-32039023 links=577 (32038447-32039023)} => ceffcf7d786e465463c364425df100e38f0dd91216be7993e3134f8729f15c1a, anchorQueue.idx=1, next retry in 1.660512011s INFO[08-27|12:54:45.903] {32040175-32040559 links=385 (32040175-32040559)} => ea55e3ceda03290d57b3ed43b58aec5c84b5252fe9c8e0d1f8832b5da32f6249, anchorQueue.idx=4, next retry in 1.660512011s INFO[08-27|12:54:45.903] {32041519-32042095 links=577 (32041519-32042095)} => 2abbbe4e89666bc3acfd536f94379bf3c0cef53831ae170884fdbc3197bb6521, anchorQueue.idx=59, next retry in 3.020142307s INFO[08-27|12:54:45.903] {32042863-32043631 links=769 (32042863-32043631)} => a97ec8c585518b9d7a6aacd8871b47d68b6212ba746771b21ceb69c33ab2c87b, anchorQueue.idx=2, next retry in 1.660512011s INFO[08-27|12:54:45.903] {32044399-32045167 links=769 (32044399-32045167)} => e9f40577ed7e298e5380b77b4fcf853498fc5fccc5a96d6769e5286f7afce619, anchorQueue.idx=5, next retry in 1.660512011s INFO[08-27|12:54:45.903] {32046127-32046703 links=577 (32046127-32046703)} => c4aec5bc5bc914861a883e1d2d41ce7f8b7fc1aa24ab6410e8a89881c4276cdd, anchorQueue.idx=105, next retry in 3.020142307s INFO[08-27|12:54:45.903] {32047663-32048239 links=577 (32047663-32048239)} => 27a02185a9cd9948e291123c29dcb9e6e758d3a0d0d6133249fb3dee592ca29d, anchorQueue.idx=3, next retry in 1.660512011s INFO[08-27|12:54:45.903] {32048815-32049775 links=961 (32048815-32049775)} => bb507d10d95411435aa693cc75963a9fc7c51e6be9a97b0fa9d8234d21a7576d, anchorQueue.idx=6, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32050543-32051311 links=769 (32050543-32051311)} => 1ef84fd8141d101e21351e5657a8f52fa2a2945c879c9bfa94404a0304a93fff, anchorQueue.idx=13, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32052079-32052847 links=769 (32052079-32052847)} => f5235c88b5ede66a3709003c99a0b91b3833c34832178854fa1e4793c048878e, anchorQueue.idx=11, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32053807-32054383 links=577 (32053807-32054383)} => fda4d3171e506feb3bc05d805a84f6b15769b4d0edda8fba55801629980c72a3, anchorQueue.idx=7, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32054959-32055919 links=961 (32054959-32055919)} => bc678a7a4afcd6783570f2621db2ace0695c0c849079324e32a6cfb384dc6075, anchorQueue.idx=37, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32056687-32057455 links=769 (32056687-32057455)} => 4b65d9d06ef1bb9a966484864671c4310ef4e998f8b27a0c1fb3827bc28e961e, anchorQueue.idx=16, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32058223-32058991 links=769 (32058223-32058991)} => 9beb230c217e9e3b1ac6b6d7a3f8e03eece1a8f99b4e169143139cc6652b4300, anchorQueue.idx=12, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32059567-32060527 links=961 (32059567-32060527)} => 6fc2e87b6d430f574016d2a1466d6aace06bcf71a833dc4b6e601a1f01545885, anchorQueue.idx=55, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32061295-32062063 links=769 (32061295-32062063)} => e7c84f67d1e607b194ad6bed03c25ec356b5943bb9d2ebef1e96ca729a173cd1, anchorQueue.idx=36, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32064751-32065135 links=385 (32064751-32065135)} => 0a32d3a2919051a58282ddee1336171c8167f852716cf132e81f0b15b331fc6b, anchorQueue.idx=109, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32065711-32066671 links=961 (32065711-32066671)} => 16c4b1855dc554d9eee0d2093cf2bb3c33b14f05f8c0087efef66bac87e4349a, anchorQueue.idx=34, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32073583-32074351 links=769 (32073583-32074351)} => 2b14338731a81d5ec3bf5840b26e4dafb0b78d0dab2353c8665a13e522d1a531, anchorQueue.idx=40, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32075695-32075887 links=193 (32075695-32075887)} => 3c6c60cb6433162dca365503c352a0e48a5f6587ad0c567c685e79665732ce67, anchorQueue.idx=8, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32078767-32078959 links=193 (32078767-32078959)} => e2c252b1c4c84857b0411785344e9ae7da38efbfffc98523cc66c17c5661daa8, anchorQueue.idx=15, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32079919-32080495 links=577 (32079919-32080495)} => d2cea2dec82007ef5902c99d9ec625e98b05e18c53eb0d3f2812ccca1a96b748, anchorQueue.idx=14, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32081071-32082031 links=961 (32081071-32082031)} => 805646512c2dd3527d8a6d8de6fdecdc73034b3915c934e5695273f4394dbb0c, anchorQueue.idx=9, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32082991-32083567 links=577 (32082991-32083567)} => 4b9197f1f87f5697f9d7c230e58fe3f9c726f343c1ff91c0d5acdcc0426ffd02, anchorQueue.idx=33, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32084527-32085103 links=577 (32084527-32085103)} => 868b8bef8a26cd49bf1183c75b98cbf558acc73feeac74f4bc5bf81c7a935234, anchorQueue.idx=29, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32087407-32088175 links=769 (32087407-32088175)} => 11210169e7a30e1c58d3918ba81a25df909b224b2624e42b5d174b0bfbf6e4bc, anchorQueue.idx=89, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32089327-32089711 links=385 (32089327-32089711)} => 5967d7cff25b0486593f8e6c18412d63646ecc6356cb20ce38083f73a9457578, anchorQueue.idx=69, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32093935-32094319 links=385 (32093935-32094319)} => c4196b740e22591f7f6be7e79515fd322e082f74d57f78792921d8afd10dfecd, anchorQueue.idx=96, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32095279-32095855 links=577 (32095279-32095855)} => 86a1d5edc30acdaec5df828aa55620e22614f832e67a0bec82f2c923a1a2668f, anchorQueue.idx=19, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32097007-32097391 links=385 (32097007-32097391)} => 066afc3e58fe8411b88a5737194a606e7bcf961f6a3417033da312c9c8476500, anchorQueue.idx=121, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32098159-32098927 links=769 (32098159-32098927)} => d0e969c04bb9052a05e398eaadb302c757ecac943ade45a6bb8796bfbb464cab, anchorQueue.idx=10, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32100079-32100463 links=385 (32100079-32100463)} => cb9568d1453770cc423b2300e98b0619042a3187d4b425d70b2e441386284655, anchorQueue.idx=99, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32103151-32103535 links=385 (32103151-32103535)} => 85761c4a1539a36c5f1d0f58447c42d00261ca8597b62edb691aaeaff27d9a17, anchorQueue.idx=39, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32104879-32105071 links=193 (32104879-32105071)} => 2902dd8bdca73f367e2cbca5ca2a31cd5a0a5825eddc3a66c9b7f079968e9408, anchorQueue.idx=23, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32106415-32106607 links=193 (32106415-32106607)} => bcc70fd8ae7314e0043aaab393708bf24b136426da3e959e90aa7a7ae96ba119, anchorQueue.idx=125, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32107951-32108143 links=193 (32107951-32108143)} => bcd465d536619b87bf336f6903ba77e95092d443ad54ac4f4fe00cbc80e0f9b6, anchorQueue.idx=17, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32109295-32109679 links=385 (32109295-32109679)} => 20b3cdc6c827dc04f8b05e3615cc5f1e62c1f4c37b352ee5443deaaf8886ffee, anchorQueue.idx=79, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32111023-32111215 links=193 (32111023-32111215)} => b6e0a827d8c81a638199649b1f1588c21394ecd148ed912889f8106ef0958130, anchorQueue.idx=61, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32114095-32114287 links=193 (32114095-32114287)} => c0be9e44c14ebcdfb6a55b957b77f16670ee64cd81c297e166f524f1d43d870b, anchorQueue.idx=128, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32115631-32115823 links=193 (32115631-32115823)} => 52a8aaa9553aed8de298cdcab8db67ae762633570d73e02aefa16c99205d475a, anchorQueue.idx=133, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32116591-32117359 links=769 (32116591-32117359)} => deb854712bc6b880624279c0004ad9aad3cab0b6fdbbd71f086856c44c00724d, anchorQueue.idx=130, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32118127-32118895 links=769 (32118127-32118895)} => 6700a80fbdac6f4c3970b884a8480401f07582b7d0e7ab1d26dac252d7538f35, anchorQueue.idx=104, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32121775-32121967 links=193 (32121775-32121967)} => 506a3e7bb63adf8e7b60e54e1f1cace155215bc3fd4c015e7382b9cd5e5c7569, anchorQueue.idx=86, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32122927-32123503 links=577 (32122927-32123503)} => fd8b94c1801414c503e1de0a7f34bb1d4bbed1d65a6cbc61f1b186b7487dda21, anchorQueue.idx=22, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32124655-32125039 links=385 (32124655-32125039)} => 1109fa0d4a74b205b45e109edc5f34df26e61434b9a68203d099b6ef2d425c8a, anchorQueue.idx=80, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32126383-32126575 links=193 (32126383-32126575)} => 96f21eb5c07b71fd4ffe746e4c064dbf53bb2f119b61caa8f3c2c31a3426153e, anchorQueue.idx=92, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32127919-32128111 links=193 (32127919-32128111)} => 6ebf566c18868d9731115e1e6c2ebb6a2f2772ea0880c04d6f58aa23fc233443, anchorQueue.idx=26, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32129455-32129647 links=193 (32129455-32129647)} => 391fadcfb87b611fc19432c0eeeec7b37db93ecd69b6c0af527995e5755697ed, anchorQueue.idx=35, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32130607-32131183 links=577 (32130607-32131183)} => 90b09276d965cdb01400d2bc555a9e57403430e1f87d1e695560cd7275722c23, anchorQueue.idx=60, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32132527-32132719 links=193 (32132527-32132719)} => 0b2479a1655e0261ecdca45f54d40c2576266bbbbc52c90081e7f1785ea9071d, anchorQueue.idx=43, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32134063-32134255 links=193 (32134063-32134255)} => 5ee1e8a265a4dff1e3bf38a194044270540589a94472e76534ccf37178b86fa4, anchorQueue.idx=28, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32135599-32135791 links=193 (32135599-32135791)} => c28836c5740138de1cd7111f89a7b5d16760123cbdb7f0a68dfb7ceac015a18c, anchorQueue.idx=132, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32141551-32141935 links=385 (32141551-32141935)} => 7f110816044a9b49982b60cb8eafefef892dda9bbd69ffca83e251da5261ef29, anchorQueue.idx=114, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32142703-32143471 links=769 (32142703-32143471)} => dacdbb1ae09f899281e9c5b13de5e6ec0a630f11369361aa4ac65b0ecf17d674, anchorQueue.idx=124, next retry in 4.375711709s INFO[08-27|12:54:45.904] {32149231-32149615 links=385 (32149231-32149615)} => 14c83f9ee8c11616845e57b1f69cf67dee4d15dbfbfb548e17e5d97c531d08c5, anchorQueue.idx=117, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32150383-32151151 links=769 (32150383-32151151)} => c584419f7175458d2a6af91fc6da9cf420e54625ef248f23e017fb167742fcbc, anchorQueue.idx=48, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32152495-32152687 links=193 (32152495-32152687)} => 631e965bb70376efd6c7717cbefb3c33df302d2572bcef29d249bca6764ca341, anchorQueue.idx=126, next retry in 3.020142307s INFO[08-27|12:54:45.904] {32154031-32154223 links=193 (32154031-32154223)} => e712284f7300b397bfe11f0f95b5b33ebf843d64cc505c801714d88391314a7b, anchorQueue.idx=51, next retry in 1.811338368s INFO[08-27|12:54:45.904] {32158063-32158831 links=769 (32158063-32158831)} => aa1882154eeebddb125359b394d7252038268995e92f6a6c7b57c66cfa803e3b, anchorQueue.idx=25, next retry in 1.660512011s INFO[08-27|12:54:45.904] {32160175-32160367 links=193 (32160175-32160367)} => a27315e49f571eba50b37bedeab1a9261b855fcca2481cf4dfed257eecdc7b03, anchorQueue.idx=68, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32161711-32161903 links=193 (32161711-32161903)} => 3c1c8c225109521c039a071bc100d02398132643139aed8aa0ebf86ed31dc17e, anchorQueue.idx=108, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32164207-32164975 links=769 (32164207-32164975)} => 8283a47645eb18ed714107b7199f151cb1d6387771162a1855490432335207b1, anchorQueue.idx=72, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32165743-32166511 links=769 (32165743-32166511)} => f857cca426d2f7bda2ab874dd7d2bf5d495070b8df77366109e0242c18eaa8fe, anchorQueue.idx=18, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32167855-32168047 links=193 (32167855-32168047)} => 1630a3dca17db6fbfc8c7adb0c7ee573933e2d6a8e7326d3634302f7d976fb51, anchorQueue.idx=111, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32169007-32169583 links=577 (32169007-32169583)} => 847b43e0909c1d820f8116d51c4eeaa6b28152fbcc975d8ef4e0f4f5b4109cee, anchorQueue.idx=42, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32170543-32171119 links=577 (32170543-32171119)} => 1595cc2da77e9e0c480453a43a527dea2492ef6df527ac43419ffd47244ac8d5, anchorQueue.idx=52, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32173615-32174191 links=577 (32173615-32174191)} => edb44d6d13bc3be7f215b6a3f841dbc05f943568fcdf1bc89107399bece9944f, anchorQueue.idx=21, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32174959-32175727 links=769 (32174959-32175727)} => 6b0a9b11684320992ebbaedd4fc7336796ea3260b626b3a1499357949c443d25, anchorQueue.idx=32, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32177071-32177263 links=193 (32177071-32177263)} => bf328b58003c18af1d6d971cd02c25cc0f3238fbce4fa9c25b7041d311167e6c, anchorQueue.idx=76, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32178031-32178799 links=769 (32178031-32178799)} => 8bbb886900a29af02ec327912230774ed889914ca19ed2fcd2a6afeeb3e62b89, anchorQueue.idx=57, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32181679-32181871 links=193 (32181679-32181871)} => 3f29b3f1a0cc672c23d4b72eaa16c0407c54ac423b747042490f5c00329189cb, anchorQueue.idx=73, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32184751-32184943 links=193 (32184751-32184943)} => 5d61d27e9b40218c83be3a17077c24fa6824e51fc2f50ffb73f0e5a8403051b4, anchorQueue.idx=112, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32185711-32186479 links=769 (32185711-32186479)} => 4210f5fc8e8fcd5e793bab7bbe9327d254b8e95808d690ea4e4bdf4f03eddff4, anchorQueue.idx=27, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32188975-32189551 links=577 (32188975-32189551)} => 883918c36f5ea0ad662148a06b0d47745eb9a36fe1041509ffd0bf520d857439, anchorQueue.idx=67, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32193583-32194159 links=577 (32193583-32194159)} => 6b9e129fadf7a6e8b52b6cb00ee43408b8ed3821242a1a6713154f6330476759, anchorQueue.idx=53, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32197039-32197231 links=193 (32197039-32197231)} => 5c227d23b2bc4776c38f28ea6c14937af4c5a65a25411339af92188677f503f3, anchorQueue.idx=98, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32198575-32198767 links=193 (32198575-32198767)} => 1036596864c5f3831b1e7b0cb356461203e2e945dc7fef967800946f602074e9, anchorQueue.idx=127, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32199919-32200303 links=385 (32199919-32200303)} => 8a14048a8cfc7d9d08e593ebcc0da79c8d6c0e26f37aad5df9d012180e052ed9, anchorQueue.idx=97, next retry in 4.375711709s INFO[08-27|12:54:45.905] {32201455-32201839 links=385 (32201455-32201839)} => f56f4c2a6cfe6fb7517ebde1e99a480d3c9015c5c710c873690e5d1431335d3b, anchorQueue.idx=56, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32202991-32203375 links=385 (32202991-32203375)} => 4014b181764881c23b629f669a6cf8985e2d024414a12b895f2d0a13c2fd60d6, anchorQueue.idx=93, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32204719-32204911 links=193 (32204719-32204911)} => d197828ca832a84ea4e025adbce168d2c69b0f3c6fb85a41f778ed4945b7f7b4, anchorQueue.idx=45, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32205679-32206447 links=769 (32205679-32206447)} => 29d27f9fbaba0668acedeeed626d9d6be277c0da56cf9048e20f57490fb78b4b, anchorQueue.idx=31, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32207599-32207983 links=385 (32207599-32207983)} => fc2b69f08aa9187565dd972ef7eaa85ada097e12d84a8b053bd886c1c1b800a8, anchorQueue.idx=70, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32210671-32211055 links=385 (32210671-32211055)} => baf68ef49754e3a92d4ff7ed41557a63e1c807dbb986af5669ffda3ce402f5a5, anchorQueue.idx=84, next retry in 4.375711709s INFO[08-27|12:54:45.905] {32212399-32212591 links=193 (32212399-32212591)} => 5d3f6be7a926437e64f9fa67fe820f636910a7d57f2612e1df76376acb8635bb, anchorQueue.idx=87, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32222767-32223343 links=577 (32222767-32223343)} => d3424b0f7080284354f011998488ef758279fc8fd532a692f9e3528871686bae, anchorQueue.idx=136, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32224495-32224879 links=385 (32224495-32224879)} => be3e8be1bb783876b0c6d3aeed387e57db0be9a8705541e7e98673dbb68c7256, anchorQueue.idx=129, next retry in 4.375711709s INFO[08-27|12:54:45.905] {32226031-32226415 links=385 (32226031-32226415)} => 1555de23e3ffe9c9f6b9cc9c857a9e3fb42afb23326ba19f96b40420ac287398, anchorQueue.idx=75, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32227567-32227951 links=385 (32227567-32227951)} => 243200befdabd31df454ffab378cc8ac6ddefaeb64a19eebbb5dd541ef9779aa, anchorQueue.idx=110, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32230639-32231023 links=385 (32230639-32231023)} => 4df1bfccba416d376ca6772539cea017e44634cffd634a6276fa334db2fac6a5, anchorQueue.idx=107, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32231791-32232559 links=769 (32231791-32232559)} => a53bfa17fed6685a1734fde5c1ea2bd8f4ce51732aeaeae3ccf22ed910613bcf, anchorQueue.idx=116, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32236975-32237167 links=193 (32236975-32237167)} => 267ddbf0c3e80bc964f328b3a6e79ec3cbcfe43bc14c4f0423af5ea9a2490d7f, anchorQueue.idx=115, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32239663-32240239 links=577 (32239663-32240239)} => 87ac20d209fde7aa3b118ee0f558df0308aa29f1fa8bdb825f06aa6f50b283f2, anchorQueue.idx=135, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32241199-32241775 links=577 (32241199-32241775)} => d9bd96f22f1939e56f6d04dba0c05dc140230e7a40c0410a8dd9b1fd120c2a09, anchorQueue.idx=137, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32242543-32243311 links=769 (32242543-32243311)} => 8cbe9fa9e24cf53eb25b16b3f225a0fe6837c8c69d09621d7df53ab3d2e01daa, anchorQueue.idx=66, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32244271-32244847 links=577 (32244271-32244847)} => 8c593c96ee6477e9817343df104f7f734f1ccf8202fb44e6c231362d26447924, anchorQueue.idx=46, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32246191-32246383 links=193 (32246191-32246383)} => f47c2c0a4819a80e681b32481d169be6c26f15b199edc3b88f587c893ad167f3, anchorQueue.idx=122, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32247343-32247919 links=577 (32247343-32247919)} => 2b5f56188cf68b70cea0801e79091d3654da4cd387cc0414eecd22ad7035a114, anchorQueue.idx=30, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32250799-32250991 links=193 (32250799-32250991)} => 7c032b90e31720228f257988deb091c25837d84f876c5360140ea5df70bd89f2, anchorQueue.idx=83, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32253679-32254063 links=385 (32253679-32254063)} => 4abed56c7b758214610aa7c657ceeb1454bf1180cf31d1b9d30ec7e68d1d00c9, anchorQueue.idx=85, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32255215-32255599 links=385 (32255215-32255599)} => 651b6fa5d3e0b8b31469e7ff1d633a6a6d91283deca37594013c0af8f669c93a, anchorQueue.idx=118, next retry in 3.020142307s INFO[08-27|12:54:45.905] {32256367-32257135 links=769 (32256367-32257135)} => e76278b8a1fcb0e00af2ae0bafd50d949ee8b600e6b9ef229fbdd92324a14578, anchorQueue.idx=24, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32258095-32258671 links=577 (32258095-32258671)} => 42085bbc73d103c4ca73257c0d1f0fe14e0f94c3294354c7f3d9e07f9cc31f70, anchorQueue.idx=38, next retry in 1.811338368s INFO[08-27|12:54:45.905] {32259439-32260207 links=769 (32259439-32260207)} => 7e05d4f8bb52b494581754b3a9f7f9c444b85eabed5c3dd3ba35fe889db6d6f7, anchorQueue.idx=138, next retry in 1.660512011s INFO[08-27|12:54:45.905] {32261359-32261743 links=385 (32261359-32261743)} => d6fd837dd99181650b9ae20b3821f4d2bfa5b4d444040dab5b60d3d2fd3c2e13, anchorQueue.idx=74, next retry in 4.375711709s INFO[08-27|12:54:45.905] {32262703-32263279 links=577 (32262703-32263279)} => c8a84ad0d895048f96012cf1e0dff24b70d1dbb96c694c34a2cf86942da07729, anchorQueue.idx=91, next retry in 3.020142307s INFO[08-27|12:54:45.906] {32266159-32266351 links=193 (32266159-32266351)} => 9e4d69359cbc5f6e41034c3bf40d68891a5238dd1c317dc77013a83902251b36, anchorQueue.idx=134, next retry in 3.020142307s INFO[08-27|12:54:45.906] {32268847-32269423 links=577 (32268847-32269423)} => accbfaf465c30d0296022e6aa9f9e9f4d568949bb9208839e73f37c107e764b1, anchorQueue.idx=119, next retry in 3.020142307s INFO[08-27|12:54:45.907] {32270383-32270959 links=577 (32270383-32270959)} => 3d8693de390300a0453306de02ac496f72e12f53b01d975ec84448da3206ec04, anchorQueue.idx=50, next retry in 1.660512011s INFO[08-27|12:54:45.907] {32271919-32272495 links=577 (32271919-32272495)} => e91b13f282b93b2e82180265d8e8505cde5f2e68ea3b73dc88838f3e980358b8, anchorQueue.idx=95, next retry in 3.174124162s INFO[08-27|12:54:45.907] {32275375-32275567 links=193 (32275375-32275567)} => 20c007362ac61689b6c44b1eaed9d7de5bcfb24edab81c95a25fffeaad3d6755, anchorQueue.idx=88, next retry in 3.020142307s INFO[08-27|12:54:45.907] {32276527-32277103 links=577 (32276527-32277103)} => 7f3065b67bf7c9da007a5878f61aebb005febe847ec9db5d03154b6de3724497, anchorQueue.idx=58, next retry in 1.660512011s INFO[08-27|12:54:45.907] {32281327-32281711 links=385 (32281327-32281711)} => a4fea00af940873ea5675ae4d06fb90f128f95453b75098c6a4c92c389a7b321, anchorQueue.idx=44, next retry in 1.660512011s INFO[08-27|12:54:45.907] {32283055-32283247 links=193 (32283055-32283247)} => 8d079a832b0cc6ef0c48e04996d3989e5bb4168961ce8ead19507a1fec8cb82c, anchorQueue.idx=78, next retry in 3.020142307s INFO[08-27|12:54:45.907] {32285935-32286319 links=385 (32285935-32286319)} => 52c8102e23a64a62b230ee5d1eaf8365d76005d6d0deb82280aa86eb44269f4d, anchorQueue.idx=100, next retry in 4.375711709s INFO[08-27|12:54:45.907] {32287663-32287855 links=193 (32287663-32287855)} => f5e3e282935b0d284b517f910da3e66c8f1fd6f786a8b03e1079c909adb5d3a8, anchorQueue.idx=82, next retry in 1.811338368s INFO[08-27|12:54:45.907] {32290735-32290927 links=193 (32290735-32290927)} => 39aa18ed2992308d803661619c196c5aee312bfc658759c2279b07f7151138dd, anchorQueue.idx=49, next retry in 1.660512011s INFO[08-27|12:54:45.907] {32291887-32292463 links=577 (32291887-32292463)} => 2130712aa8fd62f1f66a95bc5fbae2f9fd7c1eec9add47d0875ef825d82acbc1, anchorQueue.idx=90, next retry in 1.660512011s INFO[08-27|12:54:45.907] {32293807-32293999 links=193 (32293807-32293999)} => 1f6a81b38d085e75f5a7e60559f27d4ec2455e610258a5c6318cfb651b56cb83, anchorQueue.idx=123, next retry in 3.174124162s INFO[08-27|12:54:45.907] {32294959-32295535 links=577 (32294959-32295535)} => 1b11c5904a8d3a0f82dc3cc2a6cc15068da92c1b1d048749fda374ccafcd4d1f, anchorQueue.idx=113, next retry in 3.020142307s INFO[08-27|12:54:45.907] {32298415-32298607 links=193 (32298415-32298607)} => eab54f94fa5680901464a252c4c2e1a67ef645e79f169cf58dc9523728dae1a2, anchorQueue.idx=71, next retry in 3.020142307s INFO[08-27|12:54:45.907] {32299567-32300143 links=577 (32299567-32300143)} => 11d25a3a22dba56fcbe492f2a81f233be25ae43bae2e8cea9893d7ba26a92066, anchorQueue.idx=65, next retry in 1.660512011s INFO[08-27|12:54:45.907] {32301295-32301679 links=385 (32301295-32301679)} => 071f79c19ee3130774e3089a3d909f5d20886403c965925ed582fd006b73ae00, anchorQueue.idx=94, next retry in 1.660512011s INFO[08-27|12:54:45.907] {32302639-32303215 links=577 (32302639-32303215)} => c2581d9d8682d9e13ba96dbc5155fa7592120964de5d255cbef9931d6ffdb23d, anchorQueue.idx=47, next retry in 1.660512011s INFO[08-27|12:54:45.907] {32304175-32304751 links=577 (32304175-32304751)} => 979155297a5d8cc15385ee7ba7944bf036f024ae44d2e594e4b2887fd906aefd, anchorQueue.idx=120, next retry in 3.174124162s INFO[08-27|12:54:45.907] {32305903-32306287 links=385 (32305903-32306287)} => ea512a708d994e636f279cf40a45352dfe543beee1a047c912beab46baa4ccf9, anchorQueue.idx=102, next retry in 3.174124162s INFO[08-27|12:54:45.908] {32307439-32307823 links=385 (32307439-32307823)} => 7b888b7056c450d56ccd70a8afc307b9d761a8a2edc35de94a4d4b595450d16f, anchorQueue.idx=41, next retry in 1.811338368s INFO[08-27|12:54:45.908] {32310511-32310895 links=385 (32310511-32310895)} => 34624066474d8d31203c7f242b603a3862e96de4ca0e32a80f32872e90c7ce92, anchorQueue.idx=63, next retry in 1.660512011s INFO[08-27|12:54:45.908] {32315311-32315503 links=193 (32315311-32315503)} => 66ffc3212047990509478b1fabf3462dfe7ed5c0b63bca984521b2769923af2b, anchorQueue.idx=103, next retry in 3.020142307s INFO[08-27|12:54:45.908] {32316463-32317039 links=577 (32316463-32317039)} => 0db6580e305afb597a47f1ea22989c68d04b84559702b4637a5c9b3039bae0fd, anchorQueue.idx=101, next retry in 1.660512011s INFO[08-27|12:54:45.908] {32317807-32318575 links=769 (32317807-32318575)} => 4e19a6aea76396838d443ef3480c0dddd0e86869d325a8c80deb68a6ce5d69bb, anchorQueue.idx=20, next retry in 1.660512011s INFO[08-27|12:54:45.908] {32321455-32321647 links=193 (32321455-32321647)} => 2a8d41b87a1660a86b6a19a2dd89bc17dc69c5fd0f81b7ee8b12bef3b70e4957, anchorQueue.idx=81, next retry in 3.020142307s INFO[08-27|12:54:45.908] {32322799-32323183 links=385 (32322799-32323183)} => a632d4e343774c53abe9adb9b760e661fa8ac3d2356f1560a9b4f65c789c69bb, anchorQueue.idx=64, next retry in 1.660512011s INFO[08-27|12:54:45.908] {32324527-32324719 links=193 (32324527-32324719)} => 9316db97f51dca9e60a3ea069d08a9b0a1570730d486a7b21f0d339a6746f59a, anchorQueue.idx=106, next retry in 4.375711709s INFO[08-27|12:54:45.908] {32325871-32326255 links=385 (32325871-32326255)} => f64bbdc8d6410a693ed7d30e0f5a18859513e1b5c6dee23fe09dd73c25d04114, anchorQueue.idx=54, next retry in 1.660512011s INFO[08-27|12:54:45.908] {32327599-32327791 links=193 (32327599-32327791)} => dba682b617be4167601eb520fa849828058980a8ec845b344741ad0f01b028a7, anchorQueue.idx=131, next retry in 3.020142307s INFO[08-27|12:54:45.908] {32328943-32329327 links=385 (32328943-32329327)} => 84f808773cc2e7a90b057dfbbff67b298f1acc1fed8ecbee3dcd82a37b57e5e3, anchorQueue.idx=62, next retry in 1.660512011s INFO[08-27|12:54:45.908] {32388136-32388136 links=1 (32388136)} => ba3fa4dc94fa8cac0468cc83efe7132cc4e4d360890f1092aec9a38ddfc7b841, anchorQueue.idx=77, next retry in 3.020142307s ^CINFO[08-27|12:54:57.698] Got interrupt, shutting down... INFO[08-27|12:54:57.698] Exiting... INFO[08-27|12:54:57.698] Exiting Engine... INFO[08-27|12:54:57.698] RPC server shutting down INFO[08-27|12:54:57.698] RPC server shutting down INFO[08-27|12:54:57.698] RPC server shutting down WARN[08-27|12:54:57.698] Failed to serve http endpoint err="http: Server closed" WARN[08-27|12:54:57.698] Failed to serve http endpoint err="http: Server closed" INFO[08-27|12:54:57.699] HTTP endpoint closed url=0.0.0.0:8545

[donallmc]

I have the same issue when trying to sync polygon from snapshots. Looks like it was also reported in #4824 by @alexlee85 and it was concluded that it was an issue with Heimdall. I have checked my heimdall server and it appears to be healthy and it is visible to the erigon process, so I would appreciate any info about what was the specific issue with Heimdall. Also curious to know why this is now happening in an ethererum context?

[NoltaPulta]

For eth, I had to restore a backup from a week prior, there was no other way to recover.

I built a new erigon binary from alpha branch, rather than devel. I have fully caught up eth with it, and have not seen the issue resurface so far. It's possible it was a one-off error, and that devel would have been able to catch up as well, but I figured it was something to try.

Using alpha does not seem to make a difference for matic, however. I have not seen or noted any errors in the heimdall server, or the heimdall rest-server, both appear to be functioning as expected.

[NoltaPulta]

I restored matic from the initial state received from the snapshot site. Here is the log beginning stage six when the issue starts:

INFO[08-30|15:53:57.406] [6/16 Execution] Blocks execution from=32034415 to=32495395 WARN[08-30|15:54:11.974] [6/16 Execution] Execution failed block=32034432 hash=0xcd0256e2a2605018f154dc6a25dc6b201e7b7150bfb1858342d6768d22514aa4 err="bucket: TransactionLog, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position" INFO[08-30|15:54:11.974] UnwindTo block=32034431 bad_block_hash=0xcd0256e2a2605018f154dc6a25dc6b201e7b7150bfb1858342d6768d22514aa4 INFO[08-30|15:54:13.089] [6/16 Execution] Completed on block=32034431 INFO[08-30|15:54:23.029] [p2p] GoodPeers eth66=65 INFO[08-30|15:54:27.805] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.5GB sys=11.2GB INFO[08-30|15:54:33.114] [4/16 Bodies] Unwinding transactions... current block=32038873 INFO[08-30|15:54:53.115] [4/16 Bodies] Unwinding transactions... current block=32044868 INFO[08-30|15:55:13.113] [4/16 Bodies] Unwinding transactions... current block=32051692 INFO[08-30|15:55:27.782] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.2GB sys=11.2GB INFO[08-30|15:55:33.115] [4/16 Bodies] Unwinding transactions... current block=32055984 INFO[08-30|15:55:53.131] [4/16 Bodies] Unwinding transactions... current block=32060735 INFO[08-30|15:56:13.115] [4/16 Bodies] Unwinding transactions... current block=32064276 INFO[08-30|15:56:23.027] [p2p] GoodPeers eth66=65 INFO[08-30|15:56:27.779] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.0GB sys=11.2GB INFO[08-30|15:56:33.115] [4/16 Bodies] Unwinding transactions... current block=32068153 INFO[08-30|15:56:53.118] [4/16 Bodies] Unwinding transactions... current block=32073764 INFO[08-30|15:57:13.114] [4/16 Bodies] Unwinding transactions... current block=32079863 INFO[08-30|15:57:27.782] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.2GB sys=11.2GB INFO[08-30|15:57:33.118] [4/16 Bodies] Unwinding transactions... current block=32087721 INFO[08-30|15:57:53.176] [4/16 Bodies] Unwinding transactions... current block=32095339 INFO[08-30|15:58:13.116] [4/16 Bodies] Unwinding transactions... current block=32102699 INFO[08-30|15:58:23.028] [p2p] GoodPeers eth66=67 INFO[08-30|15:58:27.801] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.1GB sys=11.2GB INFO[08-30|15:58:33.114] [4/16 Bodies] Unwinding transactions... current block=32109773 INFO[08-30|15:58:53.120] [4/16 Bodies] Unwinding transactions... current block=32113519 INFO[08-30|15:59:13.114] [4/16 Bodies] Unwinding transactions... current block=32115317 INFO[08-30|15:59:29.091] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.3GB sys=11.2GB INFO[08-30|15:59:33.147] [4/16 Bodies] Unwinding transactions... current block=32116355 INFO[08-30|15:59:53.116] [4/16 Bodies] Unwinding transactions... current block=32117926 INFO[08-30|16:00:13.126] [4/16 Bodies] Unwinding transactions... current block=32119757 INFO[08-30|16:00:23.027] [p2p] GoodPeers eth66=67 INFO[08-30|16:00:27.803] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.2GB sys=11.2GB INFO[08-30|16:00:33.122] [4/16 Bodies] Unwinding transactions... current block=32121926 INFO[08-30|16:00:53.124] [4/16 Bodies] Unwinding transactions... current block=32123919 INFO[08-30|16:01:13.156] [4/16 Bodies] Unwinding transactions... current block=32125376 INFO[08-30|16:01:27.781] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.2GB sys=11.2GB INFO[08-30|16:01:33.116] [4/16 Bodies] Unwinding transactions... current block=32127068 INFO[08-30|16:01:53.161] [4/16 Bodies] Unwinding transactions... current block=32129630 INFO[08-30|16:02:13.115] [4/16 Bodies] Unwinding transactions... current block=32134165 INFO[08-30|16:02:23.027] [p2p] GoodPeers eth66=66 INFO[08-30|16:02:27.782] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.3GB sys=11.2GB INFO[08-30|16:02:33.114] [4/16 Bodies] Unwinding transactions... current block=32140018 INFO[08-30|16:02:53.121] [4/16 Bodies] Unwinding transactions... current block=32146318 INFO[08-30|16:03:13.129] [4/16 Bodies] Unwinding transactions... current block=32153626 INFO[08-30|16:03:27.782] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.2GB sys=11.2GB INFO[08-30|16:03:33.116] [4/16 Bodies] Unwinding transactions... current block=32161813 INFO[08-30|16:03:53.115] [4/16 Bodies] Unwinding transactions... current block=32170090 INFO[08-30|16:04:13.118] [4/16 Bodies] Unwinding transactions... current block=32177807 INFO[08-30|16:04:23.027] [p2p] GoodPeers eth66=67 INFO[08-30|16:04:28.628] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.3GB sys=11.2GB INFO[08-30|16:04:33.116] [4/16 Bodies] Unwinding transactions... current block=32184885 INFO[08-30|16:04:53.118] [4/16 Bodies] Unwinding transactions... current block=32191841 INFO[08-30|16:05:13.116] [4/16 Bodies] Unwinding transactions... current block=32199081 INFO[08-30|16:05:27.781] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.3GB sys=11.2GB INFO[08-30|16:05:33.116] [4/16 Bodies] Unwinding transactions... current block=32205817 INFO[08-30|16:05:53.115] [4/16 Bodies] Unwinding transactions... current block=32212497 INFO[08-30|16:06:13.118] [4/16 Bodies] Unwinding transactions... current block=32219707 INFO[08-30|16:06:23.027] [p2p] GoodPeers eth66=68 INFO[08-30|16:06:27.799] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.2GB sys=11.2GB INFO[08-30|16:06:33.114] [4/16 Bodies] Unwinding transactions... current block=32225922 INFO[08-30|16:06:53.114] [4/16 Bodies] Unwinding transactions... current block=32232440 INFO[08-30|16:07:13.176] [4/16 Bodies] Unwinding transactions... current block=32237972 INFO[08-30|16:07:27.840] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.3GB sys=11.2GB INFO[08-30|16:07:33.113] [4/16 Bodies] Unwinding transactions... current block=32243374 INFO[08-30|16:07:53.127] [4/16 Bodies] Unwinding transactions... current block=32249133 INFO[08-30|16:08:13.116] [4/16 Bodies] Unwinding transactions... current block=32254999 INFO[08-30|16:08:23.027] [p2p] GoodPeers eth66=68 INFO[08-30|16:08:27.782] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.1GB sys=11.2GB INFO[08-30|16:08:33.121] [4/16 Bodies] Unwinding transactions... current block=32260679 INFO[08-30|16:08:53.114] [4/16 Bodies] Unwinding transactions... current block=32266178 INFO[08-30|16:09:13.116] [4/16 Bodies] Unwinding transactions... current block=32270508 INFO[08-30|16:09:27.783] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.2GB sys=11.2GB INFO[08-30|16:09:33.119] [4/16 Bodies] Unwinding transactions... current block=32276332 INFO[08-30|16:09:53.125] [4/16 Bodies] Unwinding transactions... current block=32277511 INFO[08-30|16:10:13.114] [4/16 Bodies] Unwinding transactions... current block=32279311 INFO[08-30|16:10:23.027] [p2p] GoodPeers eth66=69 INFO[08-30|16:10:27.799] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.9GB sys=11.2GB INFO[08-30|16:10:33.128] [4/16 Bodies] Unwinding transactions... current block=32281886 INFO[08-30|16:10:53.156] [4/16 Bodies] Unwinding transactions... current block=32284305 INFO[08-30|16:11:13.119] [4/16 Bodies] Unwinding transactions... current block=32287092 INFO[08-30|16:11:27.785] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.2GB sys=11.2GB INFO[08-30|16:11:33.114] [4/16 Bodies] Unwinding transactions... current block=32289280 INFO[08-30|16:11:53.127] [4/16 Bodies] Unwinding transactions... current block=32290913 INFO[08-30|16:12:13.129] [4/16 Bodies] Unwinding transactions... current block=32292638 INFO[08-30|16:12:23.027] [p2p] GoodPeers eth66=69 INFO[08-30|16:12:27.800] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.1GB sys=11.2GB INFO[08-30|16:12:33.253] [4/16 Bodies] Unwinding transactions... current block=32294703 INFO[08-30|16:12:53.117] [4/16 Bodies] Unwinding transactions... current block=32299309 INFO[08-30|16:13:13.119] [4/16 Bodies] Unwinding transactions... current block=32303970 INFO[08-30|16:13:27.782] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.3GB sys=11.2GB INFO[08-30|16:13:33.115] [4/16 Bodies] Unwinding transactions... current block=32308468 INFO[08-30|16:13:53.119] [4/16 Bodies] Unwinding transactions... current block=32314609 INFO[08-30|16:14:13.120] [4/16 Bodies] Unwinding transactions... current block=32321100 INFO[08-30|16:14:23.028] [p2p] GoodPeers eth66=70 INFO[08-30|16:14:28.805] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.2GB sys=11.2GB INFO[08-30|16:14:33.115] [4/16 Bodies] Unwinding transactions... current block=32326957 INFO[08-30|16:14:53.115] [4/16 Bodies] Unwinding transactions... current block=32333150 INFO[08-30|16:15:13.120] [4/16 Bodies] Unwinding transactions... current block=32339612 INFO[08-30|16:15:27.781] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.3GB sys=11.2GB INFO[08-30|16:15:33.119] [4/16 Bodies] Unwinding transactions... current block=32346051 INFO[08-30|16:15:53.119] [4/16 Bodies] Unwinding transactions... current block=32351197 INFO[08-30|16:16:13.122] [4/16 Bodies] Unwinding transactions... current block=32358361 INFO[08-30|16:16:23.027] [p2p] GoodPeers eth66=69 INFO[08-30|16:16:27.801] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.4GB sys=11.2GB INFO[08-30|16:16:33.127] [4/16 Bodies] Unwinding transactions... current block=32361155 INFO[08-30|16:16:53.115] [4/16 Bodies] Unwinding transactions... current block=32363982 INFO[08-30|16:17:13.139] [4/16 Bodies] Unwinding transactions... current block=32366896 INFO[08-30|16:17:27.781] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.2GB sys=11.2GB INFO[08-30|16:17:33.114] [4/16 Bodies] Unwinding transactions... current block=32370209 INFO[08-30|16:17:53.118] [4/16 Bodies] Unwinding transactions... current block=32374216 INFO[08-30|16:18:13.114] [4/16 Bodies] Unwinding transactions... current block=32378311 INFO[08-30|16:18:23.027] [p2p] GoodPeers eth66=69 INFO[08-30|16:18:27.801] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.2GB sys=11.2GB INFO[08-30|16:18:33.117] [4/16 Bodies] Unwinding transactions... current block=32382409 INFO[08-30|16:18:53.119] [4/16 Bodies] Unwinding transactions... current block=32384381 INFO[08-30|16:19:13.118] [4/16 Bodies] Unwinding transactions... current block=32389376 INFO[08-30|16:19:27.782] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.3GB sys=11.2GB INFO[08-30|16:19:33.115] [4/16 Bodies] Unwinding transactions... current block=32395149 INFO[08-30|16:19:53.114] [4/16 Bodies] Unwinding transactions... current block=32401273 INFO[08-30|16:20:13.123] [4/16 Bodies] Unwinding transactions... current block=32405789 INFO[08-30|16:20:23.027] [p2p] GoodPeers eth66=71 INFO[08-30|16:20:27.809] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.5GB sys=11.2GB INFO[08-30|16:20:33.135] [4/16 Bodies] Unwinding transactions... current block=32410444 INFO[08-30|16:20:53.140] [4/16 Bodies] Unwinding transactions... current block=32414165 INFO[08-30|16:21:13.129] [4/16 Bodies] Unwinding transactions... current block=32418899 INFO[08-30|16:21:27.781] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.3GB sys=11.2GB INFO[08-30|16:21:33.120] [4/16 Bodies] Unwinding transactions... current block=32425861 INFO[08-30|16:21:53.125] [4/16 Bodies] Unwinding transactions... current block=32431637 INFO[08-30|16:22:13.116] [4/16 Bodies] Unwinding transactions... current block=32437905 INFO[08-30|16:22:23.028] [p2p] GoodPeers eth66=71 INFO[08-30|16:22:27.799] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.4GB sys=11.2GB INFO[08-30|16:22:33.127] [4/16 Bodies] Unwinding transactions... current block=32444414 INFO[08-30|16:22:53.128] [4/16 Bodies] Unwinding transactions... current block=32450103 INFO[08-30|16:23:13.114] [4/16 Bodies] Unwinding transactions... current block=32456389 INFO[08-30|16:23:27.781] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.3GB sys=11.2GB INFO[08-30|16:23:33.138] [4/16 Bodies] Unwinding transactions... current block=32463217 INFO[08-30|16:23:53.124] [4/16 Bodies] Unwinding transactions... current block=32470150 INFO[08-30|16:24:13.114] [4/16 Bodies] Unwinding transactions... current block=32476413 INFO[08-30|16:24:23.027] [p2p] GoodPeers eth66=71 INFO[08-30|16:24:28.003] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.4GB sys=11.2GB INFO[08-30|16:24:33.113] [4/16 Bodies] Unwinding transactions... current block=32482717 INFO[08-30|16:24:53.115] [4/16 Bodies] Unwinding transactions... current block=32489224 INFO[08-30|16:25:13.127] [4/16 Bodies] Unwinding transactions... current block=32494933 INFO[08-30|16:25:22.452] [4/16 Bodies] Unwind done in=31m9.339149458s INFO[08-30|16:25:27.782] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=3.5GB sys=11.2GB EROR[08-30|16:26:08.472] [9/16 IntermediateHashes] Wrong trie root of block 32034431: c15e6d0768f9b48cc54d44e3adee7730403b6d49146775f31c7155699783814d, expected (from header): e1e2479d795c44043babce00d49af482f2a4497308b133d412ddf2cd1e8e9e1b. Block hash: daebadbbcfd482de2c1e0f649a0413e8c462cf54776eca214ad8711a9396b320 WARN[08-30|16:26:08.472] Unwinding due to incorrect root hash to=32034423 INFO[08-30|16:26:08.472] UnwindTo block=32034423 bad_block_hash=0xdaebadbbcfd482de2c1e0f649a0413e8c462cf54776eca214ad8711a9396b320 INFO[08-30|16:26:08.473] [8/16 HashState] Unwinding started from=32034431 to=32034423 storage=false codes=true INFO[08-30|16:26:08.479] [8/16 HashState] Unwinding started from=32034431 to=32034423 storage=false codes=false INFO[08-30|16:26:08.489] [8/16 HashState] Unwinding started from=32034431 to=32034423 storage=true codes=false INFO[08-30|16:26:08.953] [6/16 Execution] Unwind Execution from=32034431 to=32034423 INFO[08-30|16:26:23.027] [p2p] GoodPeers eth66=67 INFO[08-30|16:26:27.753] Timings (slower than 50ms) Headers=1h25m0.774s CumulativeIndex=3.052s BlockHashes=1m14.186s Bodies=38m7.332s Senders=35m4.256s Execution=15.689s Unwind Bodies=31m9.339s Unwind Headers=24.354s HashState=9.986s IntermediateHashes=11.619s Unwind HashState=479ms Unwind Execution=675ms Unwind Senders=105ms Unwind Headers=4.161s IntermediateHashes=1.187s CallTraces=1.986s AccountHistoryIndex=2.624s StorageHistoryIndex=4.116s LogIndex=2.853s TxLookup=688ms Prune Senders=306ms INFO[08-30|16:26:27.910] [txpool] stat block=32034415 pending=10000 baseFee=30000 queued=30000 alloc=4.1GB sys=11.2GB INFO[08-30|16:26:28.162] RPC Daemon notified of new headers from=32034415 to=32495395 header sending=408.009884ms log sending=769ns INFO[08-30|16:26:28.169] [1/16 Headers] Waiting for headers... from=32034423 INFO[08-30|16:27:27.780] [txpool] stat block=32034423 pending=10000 baseFee=30000 queued=30000 alloc=6.5GB sys=11.2GB INFO[08-30|16:28:08.169] Req/resp stats req=5496 reqMin=32006299 reqMax=32416671 skel=20 skelMin=32035959 skelMax=32330679 resp=5607 respMin=32006339 respMax=32498096 dups=350465 INFO[08-30|16:28:08.188] Queue sizes anchors=2 links=416968 persisted=0 INFO[08-30|16:28:08.188] {32006492-32329335 links=322844 (32006492-32329335)} => becd33dc4b9c49e9cbc213e3686980fb506dd106be199ef8511c537390d7fa2e, anchorQueue.idx=1, next retry in 4.999561083s INFO[08-30|16:28:08.188] {32404000-32498096 links=94124 (32404000-32495405-32496224-32496243-32496448-32496644-32496645-32496654-32496660-32496662-32496663-32496667-32496678-32496694-32497514-32497515-32497516-32497517-32497518-32497519-32497520-32497521-32497522-32497523-32497921-32497922-32497951-32498096)} => c0d0c5da525aa492cec1927f11f8bf097a48a4a205d9647a41fef6786113ed37, anchorQueue.idx=0, next retry in 4.761641269s INFO[08-30|16:28:23.027] [p2p] GoodPeers eth66=67 INFO[08-30|16:28:27.805] [txpool] stat block=32034423 pending=10000 baseFee=30000 queued=30000 alloc=6.4GB sys=11.2GB INFO[08-30|16:28:28.169] Req/resp stats req=5561 reqMin=32000957 reqMax=32403807 skel=20 skelMin=32035959 skelMax=32330679 resp=5649 respMin=32001016 respMax=32498106 dups=393386 INFO[08-30|16:28:28.191] Queue sizes anchors=2 links=434800 persisted=0 INFO[08-30|16:28:28.191] {32001150-32329335 links=328186 (32001150-32329335)} => 90a242bef9936afd7ea72160f45afba07b10203cd0c4cfceda7c4ada9d839dce, anchorQueue.idx=1, next retry in 4.998785667s INFO[08-30|16:28:28.191] {32391520-32498106 links=106614 (32391520-32495405-32496224-32496243-32496448-32496644-32496645-32496654-32496660-32496662-32496663-32496667-32496678-32496694-32497514-32497515-32497516-32497517-32497518-32497519-32497520-32497521-32497522-32497523-32497921-32497922-32497951-32498106)} => 1536612d75e6d2bc098059b45250d0f714bb80b03a0181f49a685abf35b9c549, anchorQueue.idx=0, next retry in 4.913558754s ...

[NoltaPulta]

After looking through other current issues, the solution to this issue was found in #5112 by checking out commit 7dabdc3269f1e3b098fee5ab4d96ec843823a6fd from August 5th and building from there. Was able to fully catch up without issue.

That would mean that the problem commit happened sometime between August 5th and August 19th.

[michaelscheung]

This issue persist even with the latest release erigon version 2022.09.1-alpha-4067b7c4 I had to revert to erigon version 2022.99.99-dev-a02c71c0 (slightly after v2022.08.01 with VRS fix) to get it to synchronize again because of #5112

After fully synchronizing on this older version, I then updated to the aforementioned latest release because the older version has a number of other issues (such as not being able to quit in execution stage). Since then I have not been able to make any progress on synchronization because of the same issue described here. Here are the logs, which are similar to the issue described above.

[INFO] [09-09|13:28:41.638] {32853363-32853747 links=385 (32853363-32853747)} => be01c372b7471587447c475f39f59920b33e6bd3c74d150ddce5853a01f1836b, anchorQueue.idx=2, next retry in 4.937434221s 
[INFO] [09-09|13:28:41.638] {32854707-32901232 links=46526 (32854707-32901232)} => 01c22b582686950a573aad4651a9795a1a140bf5d4bff8ed3c426bbb25223fbb, anchorQueue.idx=0, next retry in 4.806169694s 
[INFO] [09-09|13:28:51.048] [txpool] stat                            block=32850675 pending=10000 baseFee=184 queued=30000 alloc=30.6GB sys=34.3GB
[INFO] [09-09|13:29:01.637] Req/resp stats                           req=1382 reqMin=32850674 reqMax=32901238 skel=18 skelMin=32852211 skelMax=33146931 resp=1470 respMin=32850675 respMax=32901240 dups=2638
[INFO] [09-09|13:29:01.639] Queue sizes                              anchors=6 links=46156 persisted=15782
[INFO] [09-09|13:29:01.639] {32852595-32853747 links=1153 (32852595-32853747)} => 4cdcbf2ed26117d32c9826fe8e7b2421bde290856b707296b27d29b817eb65d1, anchorQueue.idx=2, next retry in 4.99290009s 
[INFO] [09-09|13:29:01.639] {32854131-32856819 links=2689 (32854131-32856819)} => 1475c883af03419c1f34d700a3623ff36fcb23460be7711b37bdd44569698a84, anchorQueue.idx=4, next retry in 4.833168009s 
[INFO] [09-09|13:29:01.639] {32857011-32858355 links=1345 (32857011-32858355)} => 0dacd4343329d10e76ac2a98f567a9b78d172364abce708e56c747147e90b765, anchorQueue.idx=3, next retry in 4.768526874s 
[INFO] [09-09|13:29:01.639] {32858739-32861427 links=2689 (32858739-32861427)} => ec3ec2eeffda04b04f544df5be85aa1b9b6bc65a627776e2ac7e5368ac70b756, anchorQueue.idx=5, next retry in 4.99602093s 
[INFO] [09-09|13:29:01.639] {32862003-32898291 links=36289 (32862003-32898291)} => 744f8e17611f660e480ecbb9157ea1f2c3c87d90b86b67ce22c88e873c812a1d, anchorQueue.idx=1, next retry in 4.69237601s 
[INFO] [09-09|13:29:01.639] {32899251-32901240 links=1991 (32899251-32901239-32901240)} => 41d6120cdc12899095b7d4fef939e5d26ced2f521844115f14527deb34e7b7e9, anchorQueue.idx=0, next retry in 2.077158481s 
[INFO] [09-09|13:29:21.637] Req/resp stats                           req=1061 reqMin=32850674 reqMax=32901247 skel=19 skelMin=32852211 skelMax=33146931 resp=1132 respMin=32850675 respMax=32901248 dups=1797
[INFO] [09-09|13:29:21.639] Queue sizes                              anchors=4 links=45393 persisted=15782
[INFO] [09-09|13:29:21.639] {32852211-32852211 links=1 (32852211)} => 779dc2534467ea9925ced8b11e968054aba16ef142117e3e300fe93a7162a9fa, anchorQueue.idx=3, next retry in 4.999155567s 
[INFO] [09-09|13:29:21.639] {32853747-32853747 links=1 (32853747)} => 290d1e01bdd4a9e5b55a5409a900ef5a97bdf721b2b3168a25fe9e513ecb8962, anchorQueue.idx=2, next retry in 4.999258224s 
[INFO] [09-09|13:29:21.639] {32855091-32864499 links=9409 (32855091-32864499)} => 1ce30b2cc25e135ceb398c3e39a97da2bcdabe0adb21cbf30c69d917dc271e92, anchorQueue.idx=0, next retry in 3.063009778s 
[INFO] [09-09|13:29:21.639] {32865267-32901248 links=35982 (32865267-32901248)} => 8dc9c5559e9414f5936509b88adb8a1c54f9cf48c73be7e53694855ab0cea628, anchorQueue.idx=1, next retry in 4.831575576s 
[INFO] [09-09|13:29:41.637] Req/resp stats                           req=835 reqMin=32850674 reqMax=32901253 skel=18 skelMin=32852211 skelMax=33146931 resp=900 respMin=32850675 respMax=32901258 dups=1394
[INFO] [09-09|13:29:41.637] Queue sizes                              anchors=0 links=0 persisted=15782
[INFO] [09-09|13:29:48.798] [p2p] GoodPeers                          eth66=33
[INFO] [09-09|13:29:54.350] [txpool] stat                            block=32850675 pending=10000 baseFee=194 queued=30000 alloc=29.2GB sys=36.5GB
[INFO] [09-09|13:30:01.637] Req/resp stats                           req=1026 reqMin=32850674 reqMax=32901265 skel=19 skelMin=32852211 skelMax=33146931 resp=1136 respMin=32850675 respMax=32901268 dups=1213
[INFO] [09-09|13:30:01.639] Queue sizes                              anchors=4 links=46565 persisted=15782
[INFO] [09-09|13:30:01.639] {32852211-32852211 links=1 (32852211)} => 779dc2534467ea9925ced8b11e968054aba16ef142117e3e300fe93a7162a9fa, anchorQueue.idx=3, next retry in 4.893345388s 
[INFO] [09-09|13:30:01.639] {32853747-32853747 links=1 (32853747)} => 290d1e01bdd4a9e5b55a5409a900ef5a97bdf721b2b3168a25fe9e513ecb8962, anchorQueue.idx=2, next retry in 4.893430221s 
[INFO] [09-09|13:30:01.639] {32854131-32855283 links=1153 (32854131-32855283)} => 1475c883af03419c1f34d700a3623ff36fcb23460be7711b37bdd44569698a84, anchorQueue.idx=1, next retry in 4.878109909s 
[INFO] [09-09|13:30:01.639] {32855859-32901268 links=45410 (32855859-32901268)} => 8a4a37225c6bc69319c937baa9b47b395a372d9522bcb572aae23ee3e1668264, anchorQueue.idx=0, next retry in 4.820713309s 

[alexlee85]

@donallmc sorry for my misleading.
because the problem disappared when I switch heimdall service from localhost to heimdall.api.matic.network at that time. I'm not an expert of this project. actually, I try to resync bor nodes many times after that time, but never successed, this problem happen again and again.

[github-actions[bot]]

This issue is stale because it has been open for 40 days with no activity. Remove stale label or comment, or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 7 days with no activity.