[DRAFT] Implementing federation for logical tables in Pinot

[shauryachats]

Overview

This PR introduces federation capabilities in Apache Pinot’s broker layer, enabling a single broker to connect to and route queries across multiple independent clusters.
The change is additive and non-intrusive—it does not affect existing single-cluster functionality.

---

Key Points

• This does not break existing single-cluster setup, the new configurations pinot.secondary.cluster.names and pinot.zk.server.secondary.<cluster> enable federation support.
• The broker connects to the Helix clusters of the secondary clusters as a spectator only, to avoid showing up in the cross cluster and being included in rate-limit calculations etc.
• The federatedRoutingManager contains all the individual brokerRoutingManagers, one for each ZK cluster we connected to. It also has a getPrimaryRoutingManager to return the cluster's own routingmanager whenever the query option enableFederation is not present or not true.

[codecov-commenter]

Codecov Report

:x: Patch coverage is 21.25341% with 289 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 63.13%. Comparing base (d78b39b) to head (9debb0e). :warning: Report is 2 commits behind head on master. :white_check_mark: All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...e/pinot/broker/broker/helix/BaseBrokerStarter.java	16.25%	125 Missing and 9 partials :warning:
.../pinot/broker/routing/FederatedRoutingManager.java	0.00%	83 Missing :warning:
.../broker/routing/SecondaryBrokerRoutingManager.java	0.00%	40 Missing :warning:
...t/core/routing/CrossClusterFederationProvider.java	0.00%	12 Missing :warning:
...requesthandler/MultiStageBrokerRequestHandler.java	30.00%	6 Missing and 1 partial :warning:
.../pinot/core/routing/LogicalTableRouteProvider.java	72.72%	5 Missing and 1 partial :warning:
...sthandler/BaseSingleStageBrokerRequestHandler.java	66.66%	1 Missing and 1 partial :warning:
...e/pinot/common/utils/config/QueryOptionsUtils.java	0.00%	2 Missing :warning:
...he/pinot/common/utils/LogicalTableConfigUtils.java	83.33%	0 Missing and 1 partial :warning:
...uery/planner/physical/DispatchablePlanVisitor.java	66.66%	1 Missing :warning:
... and 1 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #17145      +/-   ##
============================================
- Coverage     63.24%   63.13%   -0.11%     
  Complexity     1432     1432              
============================================
  Files          3133     3136       +3     
  Lines        185904   186245     +341     
  Branches      28401    28449      +48     
============================================
+ Hits         117568   117585      +17     
- Misses        59277    59575     +298     
- Partials       9059     9085      +26     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-11	63.11% <21.25%> (-0.05%)	:arrow_down:
java-21	55.55% <50.79%> (-7.65%)	:arrow_down:
temurin	63.13% <21.25%> (-0.11%)	:arrow_down:
unittests	63.13% <21.25%> (-0.11%)	:arrow_down:
unittests1	55.59% <50.79%> (-0.03%)	:arrow_down:
unittests2	33.86% <16.62%> (-0.05%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ankitsultana]

@Jackie-Jiang : could we get your review on this? It's a P0 from our end so Shaurya can address feedback asap