Smarter autoscaling

[whynowy]

Summary

With current autoscaling strategy, a high throughput pipeline (e.g. lots of backlog need to process), when the pipeline processing rate hits the bottleneck (due to ISB or anything else), the replicas will still go up, unless it reaches scale.max or kind of balance, which is not expected.

We need to make the autoscaler smarter, for example, if a 5 replica vertex has similar performance to running 6 replicas, we should only do 5.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.