incubator-uniffle
incubator-uniffle copied to clipboard
[#1749] feat(remote merge): Introduce new reader for reading sorted data.
What changes were proposed in this pull request?
Introduce new reader for reading sorted data. Since #1748 already provides methods for merging blocks, we need to provide a method for reading merged block. The record obtained from the getSortedShuffleData method is sorted using the comparatorClassName which is passed by registerShuffle.
Why are the changes needed?
Fix: #1749
Does this PR introduce any user-facing change?
Yes, add doc in a separated issue
How was this patch tested?
unit test, integration test, test real job in cluster.
Codecov Report
Attention: Patch coverage is 37.05722% with 462 lines in your changes missing coverage. Please review.
Project coverage is 52.72%. Comparing base (
5ddcc28) to head (e54d40d). Report is 45 commits behind head on master.
Additional details and impacted files
@@ Coverage Diff @@
## master #2034 +/- ##
============================================
- Coverage 52.77% 52.72% -0.05%
- Complexity 2498 2667 +169
============================================
Files 398 423 +25
Lines 18135 19946 +1811
Branches 1660 1847 +187
============================================
+ Hits 9570 10516 +946
- Misses 7981 8765 +784
- Partials 584 665 +81
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Test Results
2 882 files + 75 2 882 suites +75 5h 56m 46s :stopwatch: + 3m 26s 1 019 tests + 27 1 018 :white_check_mark: + 28 1 :zzz: ±0 0 :x: ±0 12 868 runs +405 12 853 :white_check_mark: +406 15 :zzz: ±0 0 :x: ±0
Results for commit 727d072e. ± Comparison against base commit a6aefcb3.
:recycle: This comment has been updated with latest results.
@zhengchenyu Could you add documents in an another pull request?
@zhengchenyu Could you add documents in an another pull request?
Yes, I will. When the main feature are merged, I will submit a document.