rocketmq icon indicating copy to clipboard operation
rocketmq copied to clipboard

[ISSUE #8066] Tiered storage support primary/backup mode

Open bxfjb opened this issue 1 year ago • 1 comments

Which Issue(s) This PR Fixes

Fixes #8066

Brief Description

  1. 限制从节点分级存储行为 1.1. 修改 MessageStoreDispatcherImpl#dispatch:从节点不将消息分发到分级存储; 1.2. 修改 FlatFileStore#load:从节点不进行 recover 操作;
  2. 主从节点数据同步 2.1. 在SlaveSynchronize#syncAll 中增加 syncTieredMetadata方法,定时同步分级存储元数据; 2.2. 在 BrokerController 中增加 SyncTieredIndexService,同步DispatchRequest以使得从节点可以构建IndexFile
  3. 从节点切主时执行数据恢复 3.1 在TieredMessageStore中增加接口RecoverService,负责切主时恢复MetadataFlatFileIndexFile;若冷存中文件信息与本地元数据出现不一致,以冷存为准;
  4. 其他 4.1. 修改部分日志,使其更为合理; 4.2. 为防止切主时位点出现错误,FlatCommitLogFile构造函数中不再初始化位点,理论上不影响主节点构造FlatMessageFile

How Did You Test This Change?

bxfjb avatar May 29 '24 02:05 bxfjb

Codecov Report

Attention: Patch coverage is 29.16667% with 255 lines in your changes are missing coverage. Please review.

Project coverage is 42.80%. Comparing base (9c8fdb7) to head (476d286).

Files Patch % Lines
.../rocketmq/tieredstore/index/IndexStoreService.java 32.22% 51 Missing and 10 partials :warning:
...rocketmq/broker/tiered/SyncTieredIndexService.java 0.00% 52 Missing :warning:
...ocketmq/broker/processor/AdminBrokerProcessor.java 0.00% 39 Missing and 1 partial :warning:
...apache/rocketmq/broker/slave/SlaveSynchronize.java 5.55% 17 Missing :warning:
...org/apache/rocketmq/broker/out/BrokerOuterAPI.java 0.00% 14 Missing :warning:
...tmq/tieredstore/metadata/DefaultMetadataStore.java 0.00% 14 Missing :warning:
...pache/rocketmq/tieredstore/TieredMessageStore.java 50.00% 7 Missing and 2 partials :warning:
...rocketmq/tieredstore/index/DispatchRequestExt.java 35.71% 9 Missing :warning:
...a/org/apache/rocketmq/broker/BrokerController.java 0.00% 3 Missing and 3 partials :warning:
.../tieredstore/index/SyncTieredIndexRequestBody.java 0.00% 6 Missing :warning:
... and 9 more
Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8218      +/-   ##
=============================================
- Coverage      42.93%   42.80%   -0.13%     
- Complexity     10377    10394      +17     
=============================================
  Files           1270     1276       +6     
  Lines          88727    89029     +302     
  Branches       11407    11448      +41     
=============================================
+ Hits           38095    38109      +14     
- Misses         45936    46191     +255     
- Partials        4696     4729      +33     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar May 29 '24 02:05 codecov-commenter

This PR is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs. If you wish not to mark it as stale, please leave a comment in this PR.

github-actions[bot] avatar May 30 '25 00:05 github-actions[bot]

This PR was closed because it has been inactive for 3 days since being marked as stale.

github-actions[bot] avatar Jun 02 '25 00:06 github-actions[bot]