alluxio
alluxio copied to clipboard
Add TIKV as a metadata storage method
What changes are proposed in this pull request?
Add Tikv as a new metadata storage method.
Why are the changes needed?
1.To cope with the huge daily growth of data.
Thank you for your contribution. Can you add some illustrative information about the design? Have you done any actual testing?
Automated checks report:
- PR title follows the conventions: FAIL
- The title of the PR does not pass all the checks. Please fix the following issues:
- First word of title ("Adding") is not an imperative verb. Please use one of the valid words
- The title of the PR does not pass all the checks. Please fix the following issues:
- Commits associated with Github account: PASS
Some checks failed. Please fix the reported issues and reply
alluxio-bot, check this please
to re-run checks.
Automated checks report:
- PR title follows the conventions: PASS
- Commits associated with Github account: PASS
All checks passed!
Thank you for your contribution. Can you add some illustrative information about the design? Have you done any actual testing?
Thank you very much for raising questions about the code, which can help me better examine and improve the quality of code writing.
We believe that using Racks as a metastore will be a bottleneck in future large-scale application scenarios when using Alluxio. Therefore, we consider using distributed KV storage as a new metastore that can rival racks while achieving higher scalability. At present, our testing metadata has reached a scale of billions and will continue to increase in the future. We are currently only storing file meta and block meta in TIKV, hoping to receive support and assistance from the community to better improve these features.
The entire PR is only a part of the complete functionality. I will submit a total of 8 PRs, which is the first one. In the final PR, I will modify the POM files at all levels and officially integrate them into Alluxio.
Thanks for your contribution, TiKV is a good try. But I hope you can modify the code and clearly describe the design. It is best to add some tests.
This PR is only a small part of the entire feature, and we have split the entire feature for submission. At present, our functional testing has come to an end, and the data scale can reach billions.We only store file information and block information on TikV.
Using tikv as inode store can be a big feature. It definitely requires more detailed discussion before merging the code