helix
helix copied to clipboard
Creating emergency rebalance pipeline
This is a WIP PR for creating the emergency rebalance pipeline. NOTE: this is not intended to be merged, but will be used for benchmarking performance.
Logic flow: Case 1: when a rebalance is triggered, and there's no illegal placement (no new resources, no node down):
- Emergency rebalance pipeline is triggered.
- Emergency rebalance triggers partial rebalance if no partial rebalance exists; else it does nothing.
- Partial rebalance triggers asynchronous calculation.
- Emergency rebalance returns the existing Best Possible assignment.
- If partial rebalance calculates a best possible assignment that's different from the existing one, it persists the result into ZooKeeper, and triggers the pipeline again; else it does nothing.
Case 2: when a rebalance is triggered, and there are illegal placements:
- Emergency rebalance pipeline is triggered.
- Emergency rebalance assigns all illegal placements correctly, cancels existing partial rebalance thread, and persists the result to ZooKeeper.
- Emergency rebalance triggers partial rebalance.
- If partial rebalance calculates a best possible assignment that's different from the existing one, it persists the result into ZooKeeper, and triggers the pipeline again; else it does nothing.