[Feature] Support data ingestion for range distribution table
What I'm doing:
Implement end-to-end range-based routing for range distribution tables on both FE and BE sides:
FE:
Add TabletRange and Tuple.toThrift / Variant.toThrift to serialize tablet ranges as TTabletRange/TVariant. OlapTableSink / FrontendServiceImpl now populate range_distributed_columns and per-tablet range in TOlapTablePartitionParam and TOlapTableIndexTablets, and temporarily disable colocate MV index for RANGE distribution tables.
BE:
Introduce RangeRouter to route rows to tablets based on TTabletRange, supporting multi-column ranges and various open/closed/±inf intervals; Extend TabletSinkSender with RangeTabletSinkSender.
Tests:
Add/extend FE tests (VariantTest, TabletRangeTest) to cover numeric, string, boolean and date/datetime serialization to Thrift. Add BE tests (RangeRouterTest, TabletSinkSenderRangeTest) covering typical routing paths, boundary behavior, sparse selections and error cases.
Fixes #64986
What type of PR is this:
- [ ] BugFix
- [x] Feature
- [ ] Enhancement
- [ ] Refactor
- [ ] UT
- [ ] Doc
- [ ] Tool
Does this PR entail a change in behavior?
- [ ] Yes, this PR will result in a change in behavior.
- [x] No, this PR will not result in a change in behavior.
If yes, please specify the type of change:
- [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
- [ ] Parameter changes: default values, similar parameters but with different default values
- [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
- [ ] Feature removed
- [ ] Miscellaneous: upgrade & downgrade compatibility, etc.
Checklist:
- [x] I have added test cases for my bug fix or my new feature
- [ ] This pr needs user documentation (for new or modified features or behaviors)
- [ ] I have added documentation for my new feature or new function
- [ ] This is a backport pr
Bugfix cherry-pick branch check:
- [x] I have checked the version labels which the pr will be auto-backported to the target branch
- [ ] 4.0
- [ ] 3.5
- [ ] 3.4
- [ ] 3.3
[!NOTE] Implements range-based row routing for range-distributed tables, adding FE/BE support, Thrift/Proto range serialization, and comprehensive tests.
- Backend (BE):
- Add
RangeRouterandRangeTabletSinkSenderto route rows byTTabletRange; wire intoOlapTableSinkwhenis_range_distribution().- Enhance
OlapTablePartitionParamto carrydistribution_type; compute hashes only for HASH; randomize for RANGE/RANDOM.- Update build/tests; add
range_router_testandtablet_sink_sender_range_test.- Frontend (FE):
- Introduce
TabletRange,Tuple.toThrift(), andVariant.toThrift()(BOOL/INT/LARGEINT/STRING/DATE) to serialize tablet ranges.OlapTableSink/FrontendServiceImpl: populatedistribution_type, range-distributed columns, and per-tabletrangeinTOlapTablePartitionParam/TOlapTableIndexTablets; disable colocate MV for RANGE.- Add helpers in
MetaUtilsfor range distribution columns/ids;Tabletnow holdsTabletRange.- Add tests
VariantTest,TabletRangeTest; adjust existing tests for distribution info.- Thrift/Proto:
- Add
TOlapTableDistributionTypeanddistribution_typetoTOlapTablePartitionParam.- Add
TOlapTableTablet.rangeandTOlapTableIndexTablets.tablets.TTabletRangenow includes bounds and inclusivity;TVariant/VariantPBuselong_valueandstring_valuefields.Written by Cursor Bugbot for commit a007d96e43ac3bcf2705f4b3868a30e9d4fa9863. This will update automatically on new commits. Configure here.
@cursor review
@cursor review
🧪 CI Insights
Here's what we observed from your CI run for a007d96e.
🟢 All jobs passed!
But CI Insights is watching 👀
@cursor review
@cursor review
@cursor review
@cursor review
🚨 Bugbot couldn't run
Something went wrong. Try again by commenting "Cursor review" or "bugbot run", or contact support (requestId: serverGenReqId_b6e0ba7e-f4b8-410c-9135-5955b40f212f).
@cursor review
@cursor review
@cursor review
@cursor review
[BE Incremental Coverage Report]
:x: fail : 0 / 12 (00.00%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | src/exec/tablet_info.cpp | 0 | 4 | 00.00% | [276, 278, 280, 281] |
| :large_blue_circle: | src/exec/tablet_sink_sender.cpp | 0 | 2 | 00.00% | [49, 50] |
| :large_blue_circle: | src/exec/tablet_sink_sender.h | 0 | 2 | 00.00% | [72, 73] |
| :large_blue_circle: | src/exec/tablet_sink.cpp | 0 | 4 | 00.00% | [322, 324, 325, 326] |
@cursor review
@cursor review
[Java-Extensions Incremental Coverage Report]
:white_check_mark: pass : 0 / 0 (0%)
[FE Incremental Coverage Report]
:x: fail : 54 / 69 (78.26%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | com/starrocks/catalog/LargeIntVariant.java | 0 | 4 | 00.00% | [62, 63, 64, 65] |
| :large_blue_circle: | com/starrocks/sql/common/MetaUtils.java | 0 | 6 | 00.00% | [249, 254, 259, 263, 267, 271] |
| :large_blue_circle: | com/starrocks/catalog/Tablet.java | 1 | 4 | 25.00% | [41, 42, 48] |
| :large_blue_circle: | com/starrocks/planner/OlapTableSink.java | 11 | 12 | 91.67% | [517] |
| :large_blue_circle: | com/starrocks/catalog/TabletRange.java | 11 | 12 | 91.67% | [30] |
| :large_blue_circle: | com/starrocks/catalog/BoolVariant.java | 4 | 4 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/catalog/IntVariant.java | 4 | 4 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/catalog/Tuple.java | 3 | 3 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/catalog/DateVariant.java | 4 | 4 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/catalog/StringVariant.java | 4 | 4 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/service/FrontendServiceImpl.java | 12 | 12 | 100.00% | [] |
@cursor review
