base-org
base node slow sync
Hello everyone! I’m facing an issue with slow blockchain sync on a mainnet base node.
I deployed the base node on-premise with the following configuration:
Disks: RAID-5 (NVME SSD INTEL SSDPF2KX153T) VM: 16 vCPUs and 32 GB RAM I downloaded the full mainnet snapshot for the base and started the node with the snapshot data. However, the sync process is very slow, and I noticed the ETA keeps increasing.
INFO [12-23|07:38:31.127] Imported new potential chain segment number=23,836,828 hash=48bcc7..bbb4f8 blocks=1 txs=172 mgas=52.506 elapsed=6.721s mgasps=7.811 age=5d12h21m snapdiffs=397.35MiB triedirty=0.00B
WARN [12-23|07:38:39.631] Ignoring already known beacon payload number=23,836,828 hash=48bcc7..bbb4f8 age=5d12h21m
INFO [12-23|07:38:39.640] Chain head was updated number=23,836,828 hash=48bcc7..bbb4f8 root=7c64dc..6593ba elapsed=7.781567ms age=5d12h21m
INFO [12-23|07:38:44.427] Starting work on payload id=0x03170834000a8ac0
INFO [12-23|07:38:48.900] Imported new potential chain segment number=23,836,829 hash=aa9b64..5cdef9 blocks=1 txs=140 mgas=27.996 elapsed=4.454s mgasps=6.285 age=5d12h22m snapdiffs=397.40MiB triedirty=0.00B
INFO [12-23|07:38:48.908] Chain head was updated number=23,836,829 hash=aa9b64..5cdef9 root=54ed48..cd89bc elapsed=6.00817ms age=5d12h22m
INFO [12-23|07:38:56.109] Starting work on payload id=0x03f54a8b2e39dfde
INFO [12-23|07:39:10.298] Imported new potential chain segment number=23,836,830 hash=6e427b..1d5643 blocks=1 txs=188 mgas=39.307 elapsed=6.184s mgasps=6.356 age=5d12h22m snapdiffs=397.45MiB triedirty=0.00B
WARN [12-23|07:39:19.201] Ignoring already known beacon payload number=23,836,830 hash=6e427b..1d5643 age=5d12h22m
INFO [12-23|07:39:19.210] Chain head was updated number=23,836,830 hash=6e427b..1d5643 root=319e91..a2ed18 elapsed=7.384527ms age=5d12h22m
INFO [12-23|07:39:24.591] Starting work on payload id=0x032f4a6d94c7eaeb
INFO [12-23|07:39:39.271] Imported new potential chain segment number=23,836,831 hash=0af3d2..c245b2 blocks=1 txs=179 mgas=34.157 elapsed=4.757s mgasps=7.179 age=5d12h22m snapdiffs=397.51MiB triedirty=0.00B
INFO [12-23|07:39:39.282] Chain head was updated number=23,836,831 hash=0af3d2..c245b2 root=6e1885..18b071 elapsed=8.379856ms age=5d12h22m
INFO [12-23|07:39:45.404] Starting work on payload id=0x031dd651d0308981
INFO [12-23|07:39:59.623] Imported new potential chain segment number=23,836,832 hash=1174e7..75e904 blocks=1 txs=153 mgas=26.762 elapsed=5.203s mgasps=5.143 age=5d12h23m snapdiffs=397.55MiB triedirty=0.00B
WARN [12-23|07:40:09.536] Ignoring already known beacon payload number=23,836,832 hash=1174e7..75e904 age=5d12h23m
INFO [12-23|07:40:09.544] Chain head was updated number=23,836,832 hash=1174e7..75e904 root=94aaa8..56ef86 elapsed=6.444021ms age=5d12h23m
INFO [12-23|07:40:13.733] Starting work on payload id=0x03b7c289509d104a
INFO [12-23|07:40:17.789] Imported new potential chain segment number=23,836,833 hash=6405a8..4ff9a8 blocks=1 txs=154 mgas=29.458 elapsed=4.037s mgasps=7.297 age=5d12h23m snapdiffs=397.61MiB triedirty=0.00B
INFO [12-23|07:40:17.796] Chain head was updated number=23,836,833 hash=6405a8..4ff9a8 root=90d2a8..21ff18 elapsed=5.119713ms age=5d12h23m
INFO [12-23|07:40:26.440] Starting work on payload id=0x0322362d71750d50
INFO [12-23|07:40:40.512] Imported new potential chain segment number=23,836,834 hash=1e14f3..d85679 blocks=1 txs=171 mgas=39.376 elapsed=7.629s mgasps=5.161 age=5d12h23m snapdiffs=397.66MiB triedirty=0.00B
WARN [12-23|07:40:48.122] Ignoring already known beacon payload number=23,836,834 hash=1e14f3..d85679 age=5d12h23m
INFO [12-23|07:40:48.130] Chain head was updated number=23,836,834 hash=1e14f3..d85679 root=fd8224..bfc704 elapsed=6.108157ms age=5d12h23m
INFO [12-23|07:40:52.344] Starting work on payload id=0x0345731952208713
INFO [12-23|07:40:56.825] Imported new potential chain segment number=23,836,835 hash=bfba79..12e183 blocks=1 txs=154 mgas=29.979 elapsed=4.460s mgasps=6.721 age=5d12h23m snapdiffs=397.70MiB triedirty=0.00B
INFO [12-23|07:40:56.832] Chain head was updated number=23,836,835 hash=bfba79..12e183 root=e8fe03..6ff13d elapsed=5.000728ms age=5d12h23m
INFO [12-23|07:41:02.105] Starting work on payload id=0x03fa962cc29bdc3b
INFO [12-23|07:41:16.962] Imported new potential chain segment number=23,836,836 hash=d025b0..519bba blocks=1 txs=156 mgas=51.048 elapsed=5.027s mgasps=10.154 age=5d12h24m snapdiffs=397.74MiB triedirty=0.00B
WARN [12-23|07:41:27.179] Ignoring already known beacon payload number=23,836,836 hash=d025b0..519bba age=5d12h24m
INFO [12-23|07:41:27.256] Chain head was updated number=23,836,836 hash=d025b0..519bba root=70e341..ac55dc elapsed=75.999861ms age=5d12h24m
INFO [12-23|07:41:33.485] Starting work on payload id=0x03f03a08fa307c4c
INFO [12-23|07:41:49.467] Imported new potential chain segment number=23,836,837 hash=e034f0..6a1494 blocks=1 txs=178 mgas=33.738 elapsed=6.823s mgasps=4.945 age=5d12h24m snapdiffs=397.79MiB triedirty=0.00B
WARN [12-23|07:41:57.703] Ignoring already known beacon payload number=23,836,837 hash=e034f0..6a1494 age=5d12h24m
INFO [12-23|07:41:57.712] Chain head was updated number=23,836,837 hash=e034f0..6a1494 root=22e19e..073985 elapsed=7.761638ms age=5d12h24m
INFO [12-23|07:42:03.005] Starting work on payload id=0x036797e925bd08f5
INFO [12-23|07:42:17.442] Imported new potential chain segment number=23,836,838 hash=4b21fb..b59257 blocks=1 txs=183 mgas=32.270 elapsed=4.441s mgasps=7.266 age=5d12h25m snapdiffs=397.84MiB triedirty=0.00B
INFO [12-23|07:42:17.450] Chain head was updated number=23,836,838 hash=4b21fb..b59257 root=ad591b..43b6f1 elapsed=5.834872ms age=5d12h25m
INFO [12-23|07:42:23.558] Starting work on payload id=0x03475b54c9711a49
INFO [12-23|07:42:39.260] Imported new potential chain segment number=23,836,839 hash=858ce2..1d78ff blocks=1 txs=137 mgas=38.144 elapsed=6.704s mgasps=5.690 age=5d12h25m snapdiffs=397.89MiB triedirty=0.00B
WARN [12-23|07:42:47.655] Ignoring already known beacon payload number=23,836,839 hash=858ce2..1d78ff age=5d12h25m
INFO [12-23|07:42:47.665] Chain head was updated number=23,836,839 hash=858ce2..1d78ff root=d7bd0e..ad65a4 elapsed=9.011718ms age=5d12h25m
INFO [12-23|07:42:53.177] Starting work on payload id=0x0300e12a3f378477
Hello @AlexanderBarkovsky You might need to restart the whole process because this error is due to incorrect synchronization process I hope this helps
Thanks for that, but I think it won't help... I started syncing from the genesis block, but I noticed that the sync speed gradually slowed down over time. To address this, I decided to download a snapshot instead. However, it seems that the sync speed decreases on its own as time progresses.
For example, this graph shows the sync process when I started from the genesis block.
This one shows the continued sync from a snapshot.
The sync speed is not good in both cases
Increasing the cache size might help to speed up the processing of blocks
I've increased the cache size from 2048 to 8192. However, the speed is still low—around 3 requests per second.
I also added the variable OP_NODE_L1_TRUST_RPC=true.
I read somewhere that this can help increase speed if I’m using a trusted Geth node. In my case, I am using a self-hosted Geth archive node.
Have you Experiment with increasing the number of peers (--maxpeers) in your Geth configuration. This can help speed up the sync by downloading blocks and states from multiple sources simultaneously
-If you haven't already, try using the --syncmode "fast" or --syncmode "snap" options. These modes are designed to speed up the initial sync process. using Geth's pruning settings can reduce the burden on your system, allowing faster performance
Since you're using OP_NODE_L1_TRUST_RPC=true with a self-hosted Geth archive node, this setting should theoretically help. However, the performance gains can also depend on the efficiency of the archive node itself.
Given that you've increased the cache size and are still experiencing slow speeds, it might be helpful to examine the log files for any potential issues or errors that could be contributing to the slowdown.
Have you considered any of these approaches before?
We got same problem. The syncmode used in op-node is execution-layer. had similar log says Ignoring already known beacon payload and the syncing speed from geth is basically 1 block a time.
I'm wondering is it an issue mostly from geth or op-node? because it seems geth can not get a batch of recent blocks but have to discover 1 at a time.
Hello! Sorry for the late reply
Have you Experiment with increasing the number of peers (--maxpeers) in your Geth configuration. This can help speed up the sync by downloading blocks and states from multiple sources simultaneously
No, because my base node is not able to reach the maximum number of peers. For example, I only have 31 peers even when --maxpeers is set to 100.
-If you haven't already, try using the --syncmode "fast" or --syncmode "snap" options. These modes are designed to speed up the initial sync process.
The point in that I don't have the problem with leak of resources. I can increase CPU or Memory twice, but it doesn't boost the node speed. This way I expect the base should have enough resources for working properly using archive mode
However, the performance gains can also depend on the efficiency of the archive node itself.
I will try switching the Geth archive node to another one. Maybe that will help. Thank you for the advice!
Given that you've increased the cache size and are still experiencing slow speeds, it might be helpful to examine the log files for any potential issues or errors that could be contributing to the slowdown.
The strangest thing is that I don’t see any suspicious logs. Everything seems to be working correctly from the logs' perspective.
My machine's disk I/O is too low. Improving I/O and throughput should fix the slow sync issue.
@AlexanderBarkovsky hi 1- how did you get those graphs?
2- did you find any definitive answer to increasing the sync speed?
i made a small script to fetch latest block every 10 sec 5 times and then average the speed of blocks sync for me it comes out to be around 1.5-1.7 blocks per second
i am on a 6th gen xeon vm with 16vcpu and 32gb ram and 2TB nvme disk and for becon node i am using chainstack, it has a 25 api calls per sec throughput, however, i see on there dashboard the request made are less than 2 per second. which kind of matches with the number of blocks i am syncing per second
so in my theory, the node is not even attempting to fetch blocks faster
A 2tb nvme disk is not enough to sync a Base mainnet node. You need significantly more than that. Bumping up RAM and having faster disks is the best way to improve your sync speed. The issue isn't the network's sync speed but rather blocked by I/O on the hardware.
