pinot icon indicating copy to clipboard operation
pinot copied to clipboard

[federation] Add multi-cluster routing support for federated broker queries

Open shauryachats opened this issue 3 months ago • 1 comments

This PR is a part of the Cross Cluster Federation in Pinot as proposed here: https://github.com/apache/pinot/issues/16456

Overview

Introduces multi-cluster routing capability that allows a broker to query both local and remote Pinot clusters, enabling federated query execution across cluster boundaries. Subsequent PRs will hook these routing managers with the larger federation flow.

Changes

  • Added MultiClusterRoutingManager - orchestrates routing across local and remote Pinot clusters
  • Added RemoteClusterBrokerRoutingManager - manages routing for individual remote clusters with periodic sync
  • Modified BrokerRoutingManager visibility to support inheritance (5 fields/methods: private → protected)
  • Added REMOTE_CLUSTER_BROKER_ROUTING_CALCULATION_TIME_MS metric to BrokerTimer

These are new classes which are not hooked to any production flow currently, and hence do not impact any regular functionality.

Testing

Tested as a part of the draft PR: https://github.com/apache/pinot/pull/17145. Added unit tests in this PR and integration tests will be added subsequently.

shauryachats avatar Dec 01 '25 21:12 shauryachats

Codecov Report

:x: Patch coverage is 81.25000% with 24 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 63.29%. Comparing base (8fd34ec) to head (d89d815). :warning: Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
...not/broker/routing/MultiClusterRoutingManager.java 78.75% 12 Missing and 5 partials :warning:
...ker/routing/RemoteClusterBrokerRoutingManager.java 84.44% 4 Missing and 3 partials :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17296      +/-   ##
============================================
+ Coverage     63.22%   63.29%   +0.07%     
  Complexity     1474     1474              
============================================
  Files          3151     3154       +3     
  Lines        187851   187996     +145     
  Branches      28760    28779      +19     
============================================
+ Hits         118767   118991     +224     
+ Misses        59868    59787      -81     
- Partials       9216     9218       +2     
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-11 63.24% <81.25%> (+7.60%) :arrow_up:
java-21 63.23% <81.25%> (+0.02%) :arrow_up:
temurin 63.29% <81.25%> (+0.07%) :arrow_up:
unittests 63.29% <81.25%> (+0.07%) :arrow_up:
unittests1 55.68% <100.00%> (+0.02%) :arrow_up:
unittests2 33.94% <81.25%> (+0.07%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov-commenter avatar Dec 01 '25 21:12 codecov-commenter