Nethermind (Sokol Archive) stop syncing after reaching block

[MaxTeiger]

Describe the bug Nethermind (Sokol Archive) stop syncing after reaching block 21774274.

I set up a Sokol Archive node with nethermind, everything went well till the sync reached the block 21774274. Then only the following logs (see below) are displayed and nothing more is fetched from the BC.

To Reproduce Steps to reproduce the behavior:

1. Launch nethermind/nethermind with docker with these environment variables :

NETHERMIND_CONFIG: sokol_archive
NETHERMIND_SYNCCONFIG_FASTSYNC: "false"
NETHERMIND_SYNCCONFIG_FASTBLOCKS: "false"
NETHERMIND_JSONRPCCONFIG_ENABLED: "true"
NETHERMIND_JSONRPCCONFIG_HOST: 0.0.0.0
NETHERMIND_SYNCCONFIG_DOWNLOADBODIESINFASTSYNC: "true"
NETHERMIND_SYNCCONFIG_DOWNLOADRECEIPTSINFASTSYNC: "true"
NETHERMIND_PRUNINGCONFIG_MODE: "None"

Expected behavior My archive node keeps syncing to the BC and doesn't stop.

Screenshots

Logs I get :

root@POA-SOKOL-ARCHIVE-2:~/node-setup# docker logs -n 25 -f poa-sokol-archive 
2022-07-29 10:18:03.7781|Error when handling ID 1, miner_setEtherbase(0xxxxxxxxxxxxx) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_setEtherbase is not supported"},"id":1} 
2022-07-29 10:18:04.3671|Error when handling ID 1, miner_start() | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:05.8917|Error when handling ID 1, miner_start(2) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:06.3218|Error when handling ID 1, miner_start(4) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:06.9471|Error when handling ID 1, miner_start(6) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:07.4628|Error when handling ID 1, miner_start(8) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:08.0370|Error when handling ID 1, miner_start(10) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:08.8305|Error when handling ID 1, miner_start(12) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:09.3277|Error when handling ID 1, miner_start(14) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:11.0537|Error when handling ID 1, miner_start(16) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:11.7447|Error when handling ID 1, miner_start(18) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:12.2425|Error when handling ID 1, miner_start(20) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:12.9645|Error when handling ID 1, miner_start(22) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:13.3659|Error when handling ID 1, miner_start(24) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:14.1026|Error when handling ID 1, miner_start(26) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:14.8434|Error when handling ID 1, miner_start(28) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:15.1535|Error when handling ID 1, miner_start(30) | {"jsonrpc":"2.0","error":{"code":-32601,"message":"Method miner_start is not supported"},"id":1} 
2022-07-29 10:18:15.7055|Peers | with known best block: 1 | all: 1 | 
2022-07-29 10:18:30.7067|Peers | with known best block: 1 | all: 1 | 
2022-07-29 10:18:45.7107|Peers | with known best block: 1 | all: 1 | 
2022-07-29 10:19:00.7203|Peers | with known best block: 1 | all: 1 | 
2022-07-29 10:19:12.1806|Executing JSON RPC call web3_clientVersion with params [] 
2022-07-29 10:19:12.2227|Executing JSON RPC call eth_blockNumber with params [] 
2022-07-29 10:19:15.7328|Peers | with known best block: 1 | all: 1 | 
2022-07-29 10:19:30.7387|Peers | with known best block: 1 | all: 1 | 
2022-07-29 10:19:45.7424|Peers | with known best block: 1 | all: 1 | 
2022-07-29 10:19:45.7424|Sync peers - Initialized: 1 | All: 1 | Max: 50
   [      ][      ][Peer|eth66|12900000|  24.35.139.129:14136][  127|    0|    0|    0|    0|    0][Nethermind/v1.12.7-0-3b419f196-20220724/X64-Linux/6.0.3] 
2022-07-29 10:20:00.7503|Peers | with known best block: 1 | all: 1 | 

Here is configs/sokol_archive.cfg config file :

{
  "Init": {      
    "ChainSpecPath": "chainspec/sokol.json",
    "GenesisHash": "0x5b28c1bfd3a15230c9a46b399cd0f9a6920d432e85381cc6a140b06e8410112f",
    "BaseDbPath": "nethermind_db/sokol_archive",
    "LogFileName": "sokol_archive.logs.txt",
    "MemoryHint": 768000000
  },
  "TxPool": {
    "Size": 512
  },
  "EthStats": {
    "Name": "Nethermind Sokol"
  },
  "Metrics": {
    "NodeName": "Sokol Archive"
  },
  "Bloom":
  {
    "IndexLevelBucketSizes" : [16, 16, 16, 16]
  },
  "Pruning": {
    "Mode": "None"
  }    
}

Desktop:

• OS: [e.g. Windows] Docker/Debian
• Browser [e.g. chrome, safari]: N/A
• Version [e.g. 22]: 1.13.5-0-gc6b8a2498

Additional context If you have any idea on how to fix that, I'll be happy to hear it 😄

Edit: I have setup another Sokol RPC node using docker with same nethermind version, and it works perfectly well.

[tkstanczak]

Thanks @MaxTeiger -> @cbermudez97 will look at running Sokol smoke tests and will recommend you the best course of action (but he will only be online in around 3 hours).

[MaxTeiger]

Thank you very much 🙇🏼‍♂️

[cbermudez97]

Hi @MaxTeiger, thanks for the report. I'm running a node to see if I can reproduce your error with the config you shared. Still, can you share the docker command you used?

[cbermudez97]

One thing I have noticed its that the node struggles to find peers in Sokol. At least for me. Are you using bootnodes? I don't see those in your configs. Maybe also add some public archive nodes to the StaticPeers Network config.

[MaxTeiger]

Hello @cbermudez97 ! Sorry for the delay.

I tried using bootnode and without. It didn't fix my problem.

NETHERMIND_NETWORKCONFIG_BOOTNODES: "enode://10eb38a1b0384d478c98b763b5d608b79bdb1e9777391cb7fcec2d1c68f80ce2c958589e958d1d9319fd8a7713c3480444cab109091ad5cae273798dd50f9728@104.248.49.16:30303,enode://f11a0f80939b49a28bf99581da9b351a592ec1504b9d32a7dfda79b36510a891e96631239c4166e5c73368c21e9bb3241e7fd6929b899772e5a8fe9a7b7c3af6@45.77.52.149:30303"

I launched the container using docker-compose. Here is the repository w/ which I launched the node : https://github.com/MaxTeiger/node-setup

And the command I used was docker-compose -f docker-compose.poasokol-archive.yml up -d

[MaxTeiger]

Do you know where I can find some public archive nodes for Sokol ? (in order to try StaticPeers Network config)

[cbermudez97]

@MaxTeiger Thanks for the details, will review them. The resources I find about sokol testnet are the ones here: https://forum.poa.network/t/sokol-testnet-resources/1776

[MaxTeiger]

Thank you for the resources ! Do you manage to make it work ? Is it a wrong configuration on my side ?

[dceleda]

@cbermudez97 what is the status of this issue?

[MaxTeiger]

My node is still not syncing

[kamilchodola]

@cbermudez97 ping

[cbermudez97]

Sorry for the long time without updates. I'm having the same issues with the Nethermind archive I setup. So far I was unnable to find anything in the logs that leads to a clear error. Imo the main issue its that Sokol its a network without many nodes running. Nethermind easely run out of peers. For an archival node its worse since there isn't any other archive node, at least none that I'm aware of. Besides that I'm not even sure Sokol its even active. Most of the resources are from more than a year ago and there isn't any docs or resources to find around. Most of the things I found are from here but not all works.

[MaxTeiger]

Thank you for the answer @cbermudez97

There are still a few nodes which are up & running for Sokol, at least :

• https://sokol.poa.network/
• https://sokol-archive.blockscout.com/

As they are fewer and fewer, I am trying to setup at least one archive & one simple rpc, but what is weird, is that it seems that the node found peers (I set up an ethstat dashboard to follow my node sync), but doesn't process the blocks.

The only logs I've got are :

2022-10-21 13:34:34.4671|Peers | with known best block: 4 | all: 4 | 
2022-10-21 13:34:34.4671|Old Headers         0 / 27450000 | queue     0 | current     0.00bps | total     0.00bps 
2022-10-21 13:34:35.4673|Old Headers         0 / 27450000 | queue     0 | current     0.00bps | total     0.00bps 
2022-10-21 13:34:36.4670|Old Headers         0 / 27450000 | queue     0 | current     0.00bps | total     0.00bps 
2022-10-21 13:34:37.6569|Old Headers         0 / 27450000 | queue     0 | current     0.00bps | total     0.00bps 

It's the same problem if I run config sokol or sokol_archive

[MaxTeiger]

@cbermudez97 Is there a way to specify in Nethermind configuration a known node (url or ip address) on which it can synchronize ?

[cbermudez97]

@MaxTeiger Thanks for the rpcs. About your question, thats the purpose of the enode://.... And then use them as --Discovery.Bootnodes or --Network.StaticPeers. You can get a Nethermind node for its enode from logs or using:

curl http://localhost:8545 -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"net_localEnode","params":[],"id":1}'

Above rpcs enodes (all points to the same place):

# sokol.poa.network
# sokol-archive.poa.network
# sokol-archive.blockscout.com
enode://539b12a8abe4ef25dcee819f94f9bb983695aa50865364c76aa7240edca4f538ae0ab007003f7f3ded3998575a58d2c15dae700811be59b02d5a122930974869@104.248.49.16:30303

Sometimes you can find peers from other networks which will cause the node to discard them later. Without a good set of bootnodes and probably an archive it will be difficult to sync this network with full sync. So checking if its a Nethermind issue its difficult.

[MaxTeiger]

Ok thank you for your answer 😄

[asdacap]

Tried to sync. Out of 3 boot peers, only one can be pinged. Then I got invalid block on block 34.

[jmederosalvarado]

@MaxTeiger I just tried using the flags and enode that @cbermudez97 mentioned in his message above and it worked for me. For reference, set --Discovery.Bootnodes="enode://539b12a8abe4ef25dcee819f94f9bb983695aa50865364c76aa7240edca4f538ae0ab007003f7f3ded3998575a58d2c15dae700811be59b02d5a122930974869@104.248.49.16:30303"

[MaxTeiger]

@jmederosalvarado Yes it works for me too but it was very (like an eternity) for my nodes to sync. So I setup 3 archive node on the network. But it's ok for me too now ! 😄

[asdacap]

With fast sync, I can sync to head. Peer count is 6. So I'm guessing very low peer count in the node.

[asdacap]

I'm closing this issue for now. Please reopen if you found a stucked node.