[federation] Add multi-cluster routing support for federated broker queries
This PR is a part of the Cross Cluster Federation in Pinot as proposed here: https://github.com/apache/pinot/issues/16456
Overview
Introduces multi-cluster routing capability that allows a broker to query both local and remote Pinot clusters, enabling federated query execution across cluster boundaries. Subsequent PRs will hook these routing managers with the larger federation flow.
Changes
- Added
MultiClusterRoutingManager- orchestrates routing across local and remote Pinot clusters - Added
RemoteClusterBrokerRoutingManager- manages routing for individual remote clusters with periodic sync - Modified
BrokerRoutingManagervisibility to support inheritance (5 fields/methods: private → protected) - Added
REMOTE_CLUSTER_BROKER_ROUTING_CALCULATION_TIME_MSmetric toBrokerTimer
These are new classes which are not hooked to any production flow currently, and hence do not impact any regular functionality.
Testing
Tested as a part of the draft PR: https://github.com/apache/pinot/pull/17145. Added unit tests in this PR and integration tests will be added subsequently.
Codecov Report
:x: Patch coverage is 81.25000% with 24 lines in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 63.29%. Comparing base (8fd34ec) to head (d89d815).
:warning: Report is 8 commits behind head on master.
Additional details and impacted files
@@ Coverage Diff @@
## master #17296 +/- ##
============================================
+ Coverage 63.22% 63.29% +0.07%
Complexity 1474 1474
============================================
Files 3151 3154 +3
Lines 187851 187996 +145
Branches 28760 28779 +19
============================================
+ Hits 118767 118991 +224
+ Misses 59868 59787 -81
- Partials 9216 9218 +2
| Flag | Coverage Δ | |
|---|---|---|
| custom-integration1 | 100.00% <ø> (ø) |
|
| integration | 100.00% <ø> (ø) |
|
| integration1 | 100.00% <ø> (ø) |
|
| integration2 | 0.00% <ø> (ø) |
|
| java-11 | 63.24% <81.25%> (+7.60%) |
:arrow_up: |
| java-21 | 63.23% <81.25%> (+0.02%) |
:arrow_up: |
| temurin | 63.29% <81.25%> (+0.07%) |
:arrow_up: |
| unittests | 63.29% <81.25%> (+0.07%) |
:arrow_up: |
| unittests1 | 55.68% <100.00%> (+0.02%) |
:arrow_up: |
| unittests2 | 33.94% <81.25%> (+0.07%) |
:arrow_up: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.