numaflow
numaflow copied to clipboard
Autoscaling improvement for low traffic sources
Summary
When a source vertex (e.g. Kafka) has very low traffic, it might end up with a very low avg processing rate, and then a sudden backlog (pending) comes, even it's not a large number, it may still trigger a scaling up.
The essential problem is the autoscaler can not differentiate the case of low traffic caused low processing rate and being uncapable of processing.
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.