alluxio
alluxio copied to clipboard
Configure block and inode metastore separately
What changes are proposed in this pull request?
Separate configuration for block and inode is make sense, because in some cases, blockstore use ROCKS is enough, but inodestore must use HEAP, we want to distinguish the difference between these two types.
Why are the changes needed?
Sometimes we want to config inode metastore type and block metastore type seperately.
Does this PR introduce any user facing changes?
Please list the user-facing changes introduced by your change, including
- change in user-facing APIs Some configuration changes
Automated checks report:
- Commits associated with Github account: PASS
- PR title follows the conventions: FAIL
- The title of the PR does not pass all the checks. Please fix the following issues:
- First word of title ("Seperate") is not an imperative verb. Please use one of the valid words
- The title of the PR does not pass all the checks. Please fix the following issues:
Some checks failed. Please fix the reported issues and reply 'alluxio-bot, check this please' to re-run checks.
Automated checks report:
- Commits associated with Github account: PASS
- PR title follows the conventions: PASS
All checks passed!
Codecov Report
Merging #14565 (e8f64e9) into master (ea4d766) will decrease coverage by
24.87%
. The diff coverage is73.91%
.
@@ Coverage Diff @@
## master #14565 +/- ##
=============================================
- Coverage 41.83% 16.96% -24.88%
+ Complexity 9326 2702 -6624
=============================================
Files 1494 1494
Lines 87696 87841 +145
Branches 10451 10461 +10
=============================================
- Hits 36691 14900 -21791
- Misses 48021 71698 +23677
+ Partials 2984 1243 -1741
Impacted Files | Coverage Δ | |
---|---|---|
...ommon/src/main/java/alluxio/util/FeatureUtils.java | 0.00% <0.00%> (ø) |
|
...ster/src/main/java/alluxio/master/MasterUtils.java | 55.17% <66.66%> (-4.09%) |
:arrow_down: |
...common/src/main/java/alluxio/conf/PropertyKey.java | 98.07% <100.00%> (-1.38%) |
:arrow_down: |
...mon/src/main/java/alluxio/shell/CommandReturn.java | 0.00% <0.00%> (-100.00%) |
:arrow_down: |
...mon/src/main/java/alluxio/util/ExceptionUtils.java | 0.00% <0.00%> (-100.00%) |
:arrow_down: |
...n/src/main/java/alluxio/wire/AlluxioProxyInfo.java | 0.00% <0.00%> (-100.00%) |
:arrow_down: |
.../src/main/java/alluxio/wire/AlluxioMasterInfo.java | 0.00% <0.00%> (-100.00%) |
:arrow_down: |
...src/main/java/alluxio/job/meta/JobIdGenerator.java | 0.00% <0.00%> (-100.00%) |
:arrow_down: |
...n/src/main/java/alluxio/stress/BaseParameters.java | 0.00% <0.00%> (-100.00%) |
:arrow_down: |
...src/main/java/alluxio/client/UnderStorageType.java | 0.00% <0.00%> (-100.00%) |
:arrow_down: |
... and 691 more |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update ea4d766...e8f64e9. Read the comment docs.
@maobaolong Done, PTAL!
It would be better if you KEEP
MASTER_METASTORE
propertykey and if itsetByUser
, the original logic should honor the configured value and ignored theMASTER_BLOCK_METASTORE
andMASTER_INODE_METASTORE
config
Thanks for pointing this, update this pr, PTAL!
Hi, @jiacheliu3, can you take some time to review this pr?
@bzheng888 @maobaolong Can you please clarify on the use case? I'm not following. This is a very big change.
FYI: @apc999 @yuzhu @ggezer
@jiacheliu3 Thank you for your reply, the main idea for this pr is to config master inode metastore and block metastore separately. Talking about cases, if our case are big files, and the amount of inode metadata is not that much, we can config inode metastore as heap, but the block metadata will be very large, it's better to use rocks metastore to support large namespace.
@maobaolong @jiacheliu3 @dbw9580 Update this pr, PTAL!
@bzheng888 Will take a quick look soon
@bzheng888 @maobaolong Please see my previous comments above. Thx!
![image](https://user-images.githubusercontent.com/42070967/199474261-d503022f-7aa1-477c-b5da-1806c923af0b.png)
@jiacheliu3 @elega PTAL
Resolve the conflicts
alluxio-bot, merge this please