bookkeeper icon indicating copy to clipboard operation
bookkeeper copied to clipboard
verified publisher icon apache

Concurrently initialize LedgerStorage to optimize startup performance.

Open zhaizhibo opened this issue 8 months ago • 4 comments

Descriptions of the changes in this PR:

Optimize BookKeeper startup speed with multi-directory ledger configuration

Main Issue: image

Motivation

During production upgrades, BookKeeper instances with existing data typically take 3 to 5 minutes to start up.

Changes

Parallelize the initialization of DbLedgerStorage to improve startup speed.

zhaizhibo avatar Apr 21 '25 17:04 zhaizhibo

after: image image

zhaizhibo avatar Apr 22 '25 01:04 zhaizhibo

I understand that this optimization requires providing the changes before and after the optimization? Is there any verification data available?

StevenLuMT avatar Apr 24 '25 00:04 StevenLuMT

Create a big thread pool just for init storage instance is not a good idea. How many disks are you configured for one bookie?

The single-threaded configuration cannot fully utilize the performance of SSDs. Each SSD configured for a bookie node is set up with 4 or 8 directories, depending on the disk performance.

zhaizhibo avatar Jun 07 '25 03:06 zhaizhibo

Create a big thread pool just for init storage instance is not a good idea. How many disks are you configured for one bookie?

+1

gaozhangmin avatar Jul 31 '25 05:07 gaozhangmin