apache
feat: add Bulkload cu feature
What problem does this PR solve?
#1215
To solve bulk load cu problem.
Why add this feature?
In current version pegasus,as a server provider,we charge by cluster CU usage(I call it 'traditional CU in this PR'). However,when customer use bulk load feature write data to pegasus cluster,we have not charge it. Although this action uses network bandwidth and disk io.
What is changed and how does it work?
When the replicas doing bulk load , they will first download the "meta_date" file and then start download sst file,and then replicas will report these message to meta server. When meta actually bulk load success, meta can use 'bulk_load_cu_writer' to write bulk load CU usage to pegasus app ''stat''. "stat" is a special peagsus app which records traditional cu infomation.
Date structure in table stat
hashkey: data in format of "2022-10-12 00:33:25" sortkey: "bulkload_cu@appId:partitionID" value: JSON in structure of {"appId:[bulkloadCU]"}
This date structure can keep the same format with traditional CU, which is convenient for other billing system collection.
Tests
- Manual test : I test it in a pegasus cluster. It run as expected when we do a BulkLoad or we do BulkLoad on two map in the same cluster at the same time. Even we cancel BulkLoad and recover it, the program will run successfully.
FYI: BulkLoad CU will record ervery replica real downloads size.In other words,if you have to download a 100MB SST file.And 3 servers for one partition in your cluster, every partition should download 300MB total.
@ninsmiracle there are some conflicts, could you please rebase the latest master branch at first? thanks!
