The transaction volume of BSC is huge, which sometimes brings challenges for running BSC nodes with good performance. Here, information is collected and summarized for running BSC nodes. Hope it will be useful, and any suggestion or discussion is welcomed.

Binary

All the clients are suggested to upgrade to the latest release. The latest version is supposed to be more stable and better performance.

Spec for running nodes

Followings are the recommended specs for running validator and fullnode.

Running validator

2T GB of free disk space, solid-state drive(SSD), gp3, 8k IOPS, 250MB/S throughput, read latency <1ms.
12 cores of CPU and 48 gigabytes of memory (RAM).
m5zn.3xlarge instance type on AWS, or c2-standard-8 on Google cloud.
A broadband Internet connection with upload/download speeds of 10 megabyte per second

Running fullnode

2T GB of free disk space, solid-state drive(SSD), gp3, 3k IOPS, 125MB/S throughput, read latency <1ms. (if start with snap/fast sync, it will need NVMe SSD)
8 cores of CPU and 32 gigabytes of memory (RAM).
c5.4xlarge instance type on AWS, c2-standard-8 on Google cloud.
A broadband Internet connection with upload/download speeds of 5 megabyte per second.

Storage optimization

Block prune

If you do not care about the historical blocks/txs, e.g., txs in an old block, then you can take the following steps to prune blocks.

Stop the BSC node gracefully.
Run nohup geth snapshot prune-block --datadir {the data dir of your bsc node} --datadir.ancient {the ancient data dir of your bsc node} --block-amount-reserved 1024 &. It will take 3-5 hours to finish.
Start the node once the prune is done.

State prune

According to the test, the performance of a fullnode will degrade when the storage size exceeds 1.5T. We suggest the fullnode always keeps light storage by pruning the state storage.

Stop the BSC node gracefully.
Run nohup geth snapshot prune-state --datadir {the data dir of your bsc node} &. It will take 3-5 hours to finish.
Start the node once the prune is done.

Notice:

Due to that a few hours will be needed for pruning, the maintainers should always have a few backup nodes so that you can switch to the backup ones when one of them is pruning.
Prune should be taken periodically, e.g., every month, to achieve good performance.

Sync mode

Pipecommit

The pipecommit feature in release v1.1.8 for full sync. You can enable it by adding --pipecommit in the starting command when running full sync.

Light storage

When the node crashes or is force killed, the node will sync from a block that was a few minutes or a few hours ago. This is because the state in memory is not persisted into the database in real time, and the node needs to replay blocks from the last checkpoint. The replaying time dependents on the configuration TrieTimeout in the config.toml. We suggest you raise it if you can tolerate with long replaying time, so the node can keep light storage.

Performance monitoring

For importing blocks, you can monitor the following key metrics by using Prometheus/Grafana, via adding --metrics in your starting commands.

	blockInsertTimer     = metrics.NewRegisteredTimer("chain/inserts", nil) // chain_inserts in Prometheus
	blockValidationTimer = metrics.NewRegisteredTimer("chain/validation", nil) // chain_validation in Prometheus
	blockExecutionTimer  = metrics.NewRegisteredTimer("chain/execution", nil) // chain_execution in Prometheus
	blockWriteTimer      = metrics.NewRegisteredTimer("chain/write", nil) // chain_write in Prometheus

As showing in the above example, you can find more interested metrics from the source code and monitor them.

Performance tuning

In the logs, mgasps means the block processing ability of the fullnode, make sure the value is above 50.
The node can enable the profile function by adding —pprof in the starting command. Profiles can be taken by curl -sK -v http://127.0.0.1:6060/debug/pprof/profile?seconds=60 > profile_60s.out, and the dev community can help to analyze the profile.

Snapshot for new node

If you want to build a new BSC node, please fetch snapshot from bsc-snapshots.

Improvement suggestion

Feel free to raise pull requests or submit BEPs for your ideas.

References

#338
#502
official document
BEPs

Apr 26 '22 10:04 forcodedancing

Any succession for running an archive node in the cloud?

Apr 27 '22 02:04 bert2002

@forcodedancing Thanks for this content, is really useful!

Could you add the commands for running the node in different ways? (archive, light, ...) and maybe optimization tips?

Apr 27 '22 06:04 deblanco

What is faster? Diffsync or Pipecommit?

Apr 27 '22 15:04 James19903

i was prune state already , there is only 600GB in my node folder

but my node still has a liitle performance behind (compare same location and spec 's server)

i can't figure out lol

Apr 28 '22 00:04 kugimiya530

You've outlined two pruning methods. For the minimal size possible, should we be running nohup geth snapshot prune-state --datadir {the data dir of your bsc node} followed by nohup geth snapshot prune-block --datadir {the data dir of your bsc node} --datadir.ancient {the ancient data dir of your bsc node} --block-amount-reserved 1024

Is there any dependency between these two commands? For example, if we run prune-block, will we run into errors trying prune-state after?

Apr 29 '22 00:04 nathanhopp

Any succession for running an archive node in the cloud?

The disk requirement is very high. I believe a 10T ~ 15T disk is required. If you have such disk, you can have a try.