erigon icon indicating copy to clipboard operation
erigon copied to clipboard

Erigon seems stuck after committed state message

Open mauricio-seii opened this issue 3 years ago • 8 comments

Hi,

Currently, I'm trying to run an Erigon node v2022.08.22, connecting to the Ropsten network, and synchronizing from scratch. The node starts up fine but after some time processing, it seems to get stuck right after printing this logging line:

2022-08-22T17:54:05.063215506Z [INFO] [08-22|17:54:05.063] Committed State                          gas reached=549757269943 gasTarget=549755813888

So here's my question, should I leave it be, or do I need to take any action to be able to continue to process blocks?

System information

version: v2022.08.02 network: ropsten platform: Kubernetes

Expected behavior

Continue to sync to the network

mauricio-seii avatar Aug 22 '22 19:08 mauricio-seii

More logs plz

AskAlexSharov avatar Aug 23 '22 03:08 AskAlexSharov

  | [INFO] [08-22\|18:26:13.204] [p2p] GoodPeers                          eth66=46
  | [INFO] [08-22\|18:24:13.203] [p2p] GoodPeers                          eth66=46
  | [INFO] [08-22\|18:22:13.204] [p2p] GoodPeers                          eth66=46
  | [INFO] [08-22\|18:20:13.204] [p2p] GoodPeers                          eth66=46
  | [INFO] [08-22\|18:18:13.204] [p2p] GoodPeers                          eth66=46
  | [INFO] [08-22\|18:16:13.203] [p2p] GoodPeers                          eth66=45
  | [INFO] [08-22\|18:14:13.204] [p2p] GoodPeers                          eth66=45
  | [INFO] [08-22\|18:12:13.204] [p2p] GoodPeers                          eth66=45
  | [INFO] [08-22\|18:10:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|18:08:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|18:06:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|18:04:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|18:02:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|18:00:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|17:58:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|17:56:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|17:54:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|17:54:05.063] Committed State                          gas reached=549757269943 gasTarget=549755813888
  | [INFO] [08-22\|17:53:54.809] [txpool] stat                            block=12350712 pending=0 baseFee=0 queued=0 alloc=5.5GB sys=8.1GB
  | [INFO] [08-22\|17:53:49.871] [p2p] GoodPeers                          eth66=26
  | [INFO] [08-22\|17:53:47.259] [6/16 Execution] Executed blocks         number=949817 blk/s=1125.7 tx/s=2156.6 Mgas/s=1213.3 gasState=0.95 batch=933.7MB alloc=7.1GB sys=8.1GB
  | [INFO] [08-22\|17:53:27.249] [6/16 Execution] Executed blocks         number=927293 blk/s=377.4 tx/s=1041.4 Mgas/s=1270.4 gasState=0.91 batch=891.9MB alloc=4.4GB sys=8.1GB
  | [INFO] [08-22\|17:53:07.248] [6/16 Execution] Executed blocks         number=919745 blk/s=277.7 tx/s=9473.0 Mgas/s=799.7 gasState=0.86 batch=814.4MB alloc=4.7GB sys=7.8GB
  | [INFO] [08-22\|17:52:54.809] [txpool] stat                            block=12350712 pending=0 baseFee=0 queued=0 alloc=6.2GB sys=7.7GB
  | [INFO] [08-22\|17:52:47.262] [6/16 Execution] Executed blocks         number=914196 blk/s=24.2 tx/s=18619.8 Mgas/s=391.1 gasState=0.83 batch=776.9MB alloc=4.9GB sys=7.7GB
  | [INFO] [08-22\|17:52:27.253] [6/16 Execution] Executed blocks         number=913711 blk/s=50.0 tx/s=22486.9 Mgas/s=473.3 gasState=0.82 batch=775.8MB alloc=4.5GB sys=7.0GB
  | [INFO] [08-22\|17:52:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|17:52:07.253] [6/16 Execution] Executed blocks         number=912711 blk/s=814.9 tx/s=3584.7 Mgas/s=950.9 gasState=0.80 batch=774.5MB alloc=4.1GB sys=6.8GB
  | [INFO] [08-22\|17:51:54.809] [txpool] stat                            block=12350712 pending=0 baseFee=0 queued=0 alloc=5.3GB sys=6.8GB
  | [INFO] [08-22\|17:51:49.871] [p2p] GoodPeers                          eth66=23
  | [INFO] [08-22\|17:51:47.248] [6/16 Execution] Executed blocks         number=896409 blk/s=625.0 tx/s=3505.9 Mgas/s=1214.5 gasState=0.76 batch=723.4MB alloc=4.1GB sys=6.8GB
  | [INFO] [08-22\|17:51:27.253] [6/16 Execution] Executed blocks         number=883913 blk/s=1039.3 tx/s=14286.1 Mgas/s=954.9 gasState=0.72 batch=654.8MB alloc=5.9GB sys=6.6GB
  | [INFO] [08-22\|17:51:07.251] [6/16 Execution] Executed blocks         number=863124 blk/s=92.4 tx/s=15772.0 Mgas/s=386.5 gasState=0.69 batch=640.6MB alloc=3.1GB sys=6.6GB
  | [INFO] [08-22\|17:50:54.808] [txpool] stat                            block=12350712 pending=0 baseFee=0 queued=0 alloc=5.0GB sys=6.6GB
  | [INFO] [08-22\|17:50:47.247] [6/16 Execution] Executed blocks         number=861276 blk/s=1843.2 tx/s=12155.1 Mgas/s=591.5 gasState=0.67 batch=637.5MB alloc=3.5GB sys=6.6GB
  | [INFO] [08-22\|17:50:27.248] [6/16 Execution] Executed blocks         number=824413 blk/s=2642.1 tx/s=4289.5 Mgas/s=746.8 gasState=0.65 batch=619.7MB alloc=3.3GB sys=6.6GB
  | [INFO] [08-22\|17:50:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|17:50:07.248] [6/16 Execution] Executed blocks         number=771572 blk/s=1571.1 tx/s=2725.0 Mgas/s=435.3 gasState=0.62 batch=589.3MB alloc=4.0GB sys=6.6GB
  | [INFO] [08-22\|17:49:54.812] [txpool] stat                            block=12350712 pending=0 baseFee=0 queued=0 alloc=3.0GB sys=6.6GB
  | [INFO] [08-22\|17:49:49.871] [p2p] GoodPeers                          eth66=20
  | [INFO] [08-22\|17:49:47.247] [6/16 Execution] Executed blocks         number=740147 blk/s=2222.2 tx/s=2877.5 Mgas/s=716.9 gasState=0.61 batch=575.2MB alloc=4.8GB sys=6.6GB
  | [INFO] [08-22\|17:49:27.257] [6/16 Execution] Executed blocks         number=695726 blk/s=2039.5 tx/s=1943.3 Mgas/s=522.7 gasState=0.58 batch=548.9MB alloc=6.1GB sys=6.5GB
  | [INFO] [08-22\|17:49:07.257] [6/16 Execution] Executed blocks         number=654938 blk/s=2058.3 tx/s=2706.1 Mgas/s=617.8 gasState=0.56 batch=530.5MB alloc=4.3GB sys=6.2GB
  | [INFO] [08-22\|17:48:54.831] [txpool] stat                            block=12350712 pending=0 baseFee=0 queued=0 alloc=5.8GB sys=6.2GB
  | [INFO] [08-22\|17:48:47.248][6/16 Execution] Executed blocks         number=613751 blk/s=752.2 tx/s=6621.3 Mgas/s=776.5 gasState=0.54 batch=510.5MB alloc=5.0GB sys=6.0GB
  | [INFO] [08-22\|17:48:27.296] [6/16 Execution] Executed blocks         number=598743 blk/s=822.1 tx/s=5531.1 Mgas/s=973.2 gasState=0.51 batch=496.0MB alloc=4.0GB sys=6.0GB
  | [INFO] [08-22\|17:48:13.204] [p2p] GoodPeers                          eth66=44
  | [INFO] [08-22\|17:48:07.248] [6/16 Execution] Executed blocks         number=582261 blk/s=693.7 tx/s=3282.5 Mgas/s=632.3 gasState=0.48 batch=479.8MB alloc=4.1GB sys=5.7GB
  | [INFO] [08-22\|17:47:54.809] [txpool] stat                            block=12350712 pending=0 baseFee=0 queued=0 alloc=2.9GB sys=5.7GB

mauricio-seii avatar Aug 23 '22 15:08 mauricio-seii

add --pprof show output of http://localhost:6060/debug/pprof/goroutine?debug=1

AskAlexSharov avatar Aug 24 '22 02:08 AskAlexSharov

Ropsten - network already passed TheMerge event - do you use ConsensusLayer service?

AskAlexSharov avatar Aug 24 '22 02:08 AskAlexSharov

Ropsten - network already passed TheMerge event - do you use ConsensusLayer service?

No, we wanted this node to be synced with the network before connecting it to a CL client.

edit: we started a fresh instance, so our expectation was to have it processing all blocks until it got to the merge event, and then we'd change the config to connect to the CL client.

mauricio-seii avatar Aug 24 '22 19:08 mauricio-seii

I think this is not possible. @yperbasis , right?

AskAlexSharov avatar Aug 25 '22 02:08 AskAlexSharov

hang on, let me try again. We expected the node to be syncing up to around the 12m mark, and once we got notice of the merge event we'd update the config and set up the integration to a CL.

What you are saying is that, if I'm starting a new Erigon node I should immediately connect it to a CL is that correct?

mauricio-seii avatar Aug 25 '22 03:08 mauricio-seii

For networks that have already passed the Merge, it's best to drive Erigon by CL from the start. The alternative that you're describing, namely to first sync all pre-Merge blocks, relies on block gossip (at least when Erigon block snapshots end before the Merge). EIP-3675 prohibits block gossip after the Merge, rendering this strategy brittle and depending on a small fraction of peers who are left not updated with Ropsten Terminal Total Difficulty and still gossiping. Moreover, with PR #5172 such sync without a CL node will be disabled.

TL;DR: Connect your Erigon node to a CL from the start.

yperbasis avatar Aug 25 '22 06:08 yperbasis

awesome thank you very much @yperbasis and @AskAlexSharov!

mauricio-seii avatar Aug 25 '22 11:08 mauricio-seii