[QUERY] Function Drain mode not working using ServiceBusTrigger with Microsoft.Azure.WebJobs.Extensions.ServiceBus 5.7.0

[Jomatam]

Library name and version

Microsoft.Azure.WebJobs.Extensions.ServiceBus 5.7.0

Query/Question

We are running a ServiceBusTrigger function on an Azure Functions Elastic Premium plan. Until this week we used Microsoft.Azure.WebJobs.Extensions.ServiceBus 4.2.1. This week we updated to 5.7.0.

With this change we see that the drain mode/graceful shutdown of the function stopped working as before. With version 4.2.1 we almost saw no deliveryCounts > 1 (roughly one in 3000). Since the move to 5.7.0 we are seeing a 10-fold increase in occurrences of deliveryCounts > 1. In our case deliveryCounts > 1 are typically only caused by function executions being aborted as result of scale-in.

Did something change in the handling of drain mode between versions 4.2.1 and 5.7.0?

Environment

Azure Premium Elastic function on Net 6, functions v4, non-isolated

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[JoshLove-msft]

Can you please share more details of your app such as whether you are using any Startup/shutdown code? The drain feature should work the same in 5.x.

[Jomatam]

• We only run the single ServiceBusTrigger function on the function plan
• The function is a single-threaded CPU-intensive process that take at most 20 minutes.
• We set the servicebus.messageHandlerOptions.maxConcurrentCalls to be equal to the number of cores the plan has available per instance (EP3, so 4 in this case).
• We are currently not using any specific startup or shutdown code. We know the process can always finish within the maximum drain mode window, so we do not abort/cancel the process but let it finish.

This worked perfectly using version 4.2.1 of the package. Despite peaky load characteristics (and a resulting high variability in active instances, almost no processes were aborted at all.

[JoshLove-msft]

Can you share your function code?

[Jomatam]

Unfortunately, I cannot. The code contains proprietary algorithms. I will try to create a simple repro project, but that will take some time.

Would it matter that we run a C++ executable inside the function?

For now, I reverted the package upgrade back to 4.2.1. The deliveryCount immediately recovered (see deliverycount ApplicationInsights chart below, red dot is the deployment moment).

[JoshLove-msft]

I suspect there may be something else going on resulting in the increase in delivery counts. The handling in the 5.x version is essentially the same as 4.x when it comes to scale in. One thing to check would be whether you are binding to the cancellation token in your function. This token would be canceled when scaling in.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!