Tracking Issue: Automated load balancing based on real load

[ZuLiangWang]

Describe This Problem

In the roadmap https://github.com/apache/incubator-horaedb/issues/1411 we discussed recently, we need to implement load balancing based on real load this year, rather than simply scheduling based on the number of shards and tables.

Currently, all our balancing strategies are based on the number of shards and tables. However, the number of shards and tables cannot reflect the actual load of the node. In the presence of hotspot tables, our current load balancing schedule cannot avoid excessive load on some nodes.

Proposal

Design a load balancing strategy based on real load scheduling.

Collection and reporting of real loads.

• Define the indicators that HoraeDB reports to HoraeMeta.
  • These indicators must be able to reflect the actual load of the current shard and node.
  • Collect this indicators, the cost of collecting these indicators needs to be small enough so that it does not affect the performance of the service.

Scheduling based on real load

• Design a scheduling strategy based on real load in HoreaMeta
  • The scheduling strategy must be compatible and coexist with the existing scheduling based on the number of shards and tables.
  • The automatic scheduling strategy should be simple and safe enough, this is even more important than the effectiveness of scheduling.

Additional Context

No response