Support consumequeue, indexfile multiple directories storage mode

[shengminw]

In RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support, CommitLog File has supported multi-directories storage mode. In this case, the commitlog can be stored on multiple disks, but consumequeue/index is still stored on one disk. However, the size of consumequeue/index can reach tens of Gbs, or even 100 Gbs, and the space used can't be easily ignored. The current commitLog multipath needs to be configured independently, as follows:

storePathRootDir=/rmqstore1
storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog

Therefore, I optimized the multi-path mode, and the improvements are as follows: ● Add multi-path mode for consumequeue.
● Add multi-path mode for indexfile. ● Optimize the method to get the commitlog path. It is not necessary to specify storePathCommitLog separately, and generate the storePathCommitLog from storePathRootDir. ● Add a Timed thread to check path capacity. When select store path, it will try to avoid writing to a nearly full disk. ● Loading metadata in multi-path. By default, the config file is stored in the first path, and the config bak file is stored in the second path. For example: storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config. At startup, scan all paths and load the config file from the latest path of dataversion.（This ensures that when the path is changed, the latest version can be loaded） ● Loading checkpoint file and abort file in multi-path. The loading path of the checkpoint file and the abort file is the same as that of the config file.

Chinese Version / 中文版本

在RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support 当中，CommitLog文件已经支持了多文件路径的存储模式。 在这个情况下，commitLog可以被存储至多块磁盘当中，但是consumequeue和index依然存储于单盘上。然而consumequeue 和 index的文件大小也可能达到数十G或者上百G的大小。这样的存储空间并不能被忽略。 当前的commitLog的多路径需要被额外配置，如下所示：

storePathRootDir=/rmqstore1
storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog

因此，我对多路径模式做了一定的优化，改进主要如下： ● consumequeue的存储增加多路径模式。 ● indexfile的存储增加多路径模式。 ● 优化commitlog获取文件路径的方式。不必要单独设置storePathCommitLog，storePathCommitLog可以从storePathRootDir根目录下直接生成。 ● 增加一个定时线程检查路径的容量。当选择写入的路径的时候，能尽量避免写入至快满的磁盘中。 ● 多路径下的元数据加载。默认将config文件夹写入至第1个路径中，config文件夹的bak文件写入至第2个路径。例如：storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config。在启动的时候，扫描所有路径，选择dataversion最新的所在路径作为初始加载路径。（这样能确保路径被更改时，能够从最新的路径下加载） ● 加载checkpoint文件和abort文件。checkpoint文件和abort文件的加载路径与config路径保持一致。

[lizhanhui]

@shengminw In your RIP:

So the deletion should be executed when any of the disks reached the threshold (set by diskMaxUsedSpaceRatio)

Is this a good idea? Attempting to delete expiring data is pretty expensive, which may compete IOPS with normal data persistence, causing unexpected latency spikes.

Another issue: similar to https://github.com/apache/rocketmq/issues/4803 Watermarks measured in percent are inferior to those measured in kilobytes(megabytes) as Linux kernel does. The reclamation of RocketMQ should follow the same way.

[shengminw]

@lizhanhui Thanks for your suggestion, I think it is a good way that measuring in megabytes is better than the ratio. But RIP-7 is not my job, I just make further improvements to the issue. This issue hopes to expand the multi-disk paths of consumequeue, indexfile and related files on the basis of RIP-7. Maybe disk capacity measurement can be resolved in a new issue.

[lizhanhui]

@shengminw Okay, move on with your improvement...I thought you are building RIP-7.

[github-actions[bot]]

This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.

[github-actions[bot]]

This issue was closed because it has been inactive for 3 days since being marked as stale.