Trigger will not scale down as expected

[lilleman12]

When having multiple partition the trigger will scale up on high load but will not scale down on low load. The trigger will ask the instance/host to scale down but as we continusly get new messages at low rate the instance/host will think that we are still active and will not scale down. This is a confirmed bug by Microsoft in the host software and is in the roadmap somewhere.

In the mean time there is work around and that is to increase the maxBatchSize property and SubscriberIntervalInSeconds this will simulate that the trigger is inactive as it collecting messages at low load and the host will scale down.

[TsuyoshiUshio]

Thank you, @lilleman12

According to this logic, https://github.com/Azure/azure-functions-kafka-extension/blob/dev/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Listeners/KafkaTopicScaler.cs#L235

Current logic will be

Scale In happens... 1. The number of the works exceed the number of the partition 2. All partition doesn't have messages 3. Decrease trend of messages

In this case, if the burst coming, Premium plan scale up to 32 workers (the same as the number of partition). Then suddenly traffic has gone, however, keep on sending 1 message. In this case, never scale down.

The solution will be watching how many partitions have messages. https://github.com/Azure/azure-functions-kafka-extension/blob/dev/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Listeners/KafkaTopicScaler.cs#L178

At least, this one need to be improve. IMO.