alluxio icon indicating copy to clipboard operation
alluxio copied to clipboard

Add TIKV as a metadata storage method

Open wangbin83-gmail-com opened this issue 10 months ago • 6 comments

What changes are proposed in this pull request?

Add Tikv as a new metadata storage method.

Why are the changes needed?

1.To cope with the huge daily growth of data.

wangbin83-gmail-com avatar Apr 03 '24 01:04 wangbin83-gmail-com

Thank you for your contribution. Can you add some illustrative information about the design? Have you done any actual testing?

jasondrogba avatar Apr 11 '24 09:04 jasondrogba

Automated checks report:

  • PR title follows the conventions: FAIL
    • The title of the PR does not pass all the checks. Please fix the following issues:
      • First word of title ("Adding") is not an imperative verb. Please use one of the valid words
  • Commits associated with Github account: PASS

Some checks failed. Please fix the reported issues and reply alluxio-bot, check this please to re-run checks.

alluxio-bot avatar Apr 18 '24 08:04 alluxio-bot

Automated checks report:

  • PR title follows the conventions: PASS
  • Commits associated with Github account: PASS

All checks passed!

alluxio-bot avatar Apr 18 '24 08:04 alluxio-bot

Thank you for your contribution. Can you add some illustrative information about the design? Have you done any actual testing?

Thank you very much for raising questions about the code, which can help me better examine and improve the quality of code writing.

We believe that using Racks as a metastore will be a bottleneck in future large-scale application scenarios when using Alluxio. Therefore, we consider using distributed KV storage as a new metastore that can rival racks while achieving higher scalability. At present, our testing metadata has reached a scale of billions and will continue to increase in the future. We are currently only storing file meta and block meta in TIKV, hoping to receive support and assistance from the community to better improve these features.

The entire PR is only a part of the complete functionality. I will submit a total of 8 PRs, which is the first one. In the final PR, I will modify the POM files at all levels and officially integrate them into Alluxio.

wangbin83-gmail-com avatar Apr 18 '24 08:04 wangbin83-gmail-com

Thanks for your contribution, TiKV is a good try. But I hope you can modify the code and clearly describe the design. It is best to add some tests.

This PR is only a small part of the entire feature, and we have split the entire feature for submission. At present, our functional testing has come to an end, and the data scale can reach billions.We only store file information and block information on TikV.

wangbin83-gmail-com avatar May 23 '24 02:05 wangbin83-gmail-com

Using tikv as inode store can be a big feature. It definitely requires more detailed discussion before merging the code

apc999 avatar Jun 26 '24 17:06 apc999