apache
[ISSUE #5087] RocketMQ Store Module Improvement
RocketMQ Store Module Improvement. If you want more information, please visit https://github.com/apache/rocketmq/issues/5087
- [x] [Store-A] Speed up Broker initialization by concurrently loading ConsumeQueue
- [ ] [Store-B] Support automatic deletion of ConsumeQueue files without valid data to save disk space
- [ ] [Store-C] WIP
Store-A: [#5093] When the number of ConsumeQueues on the Broker is large, the startup speed is very slow and the recovery time is long. After testing, it takes about 150 seconds to "cold recover" 200,000 ConsumeQueue files in a virtual machine (16C 32G 1T SSD) environment. After using this optimization, it only takes 48 seconds, saving 2/3 of the startup time.
Store-B: There are a large number of topics (include RETRY and DLQ) that are no longer used on the Broker occupies a lot of disk space. We use scheduled tasks to delete these index files, but after deleting the ConsumeQueue, the offset will start from 0 again. There are some side effects for subscription groups that consume using pull mode.
@lizhimins @azhsmesos Has store-B been resolved?
No one assigned it to me, so I think it had been claimed by someone else and I didn't do it. sorry~
@lizhimins @azhsmesos Has store-B been resolved?
No one assigned it to me, so I think it had been claimed by someone else and I didn't do it. sorry~
@azhsmesos hhhhhhhhhhhhhhhhhh，in fact, you don’t need to wait for others to assign you, you can directly submit a PR.