numaflow icon indicating copy to clipboard operation
numaflow copied to clipboard

Autoscaling improvement for low traffic sources

Open whynowy opened this issue 1 year ago • 0 comments

Summary

When a source vertex (e.g. Kafka) has very low traffic, it might end up with a very low avg processing rate, and then a sudden backlog (pending) comes, even it's not a large number, it may still trigger a scaling up.

The essential problem is the autoscaler can not differentiate the case of low traffic caused low processing rate and being uncapable of processing.


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

whynowy avatar Dec 06 '23 21:12 whynowy