Azure function gets stuck running to 100% CPU

[andriysavin]

trafficstars

I've recently deployed a new function app, which runs on app service plan and in isolated worker mode. The function is rather simple, it has a storage queue trigger, on which it pulls a file from an Web API and copies it to an SFTP share. Once started, everything looks fine, and app service plan is very low on CPU utilization. But once the function processes at least one message, plans's CPU consumption goes about 100% and remains at that level indefinitely. Since the function is run in isolated mode, I see no way to profile the host process which I suspect. The reason I suspect it is that function app's process has very low cpu consuption after it finished processing the last message (observed in the Kudu Console's Process Explorer). This also makes function's portal page hang and fail to load function details. There are also some other app service apps running on the plan, but they have low impact on CPU consuption. After this hang happens, the function stops processing more queue messages. Once I stop it, the CPU usage drops to 0, and after starting it again everything repeats.

Investigative information

Please provide the following:

• Timestamp: 2024-03-26T13:07:34.957769Z
• Function App version: V4
• Invocation ID: 815f35e9-fa41-4868-905c-9a46a3d75cc8
• Region: Central US

Repro steps

1. Create C#/.Net isolated worker function with storage queue trigger;
2. Deploy to app service plan azure function app;
3. Add a message to the queue to trigger the function.

Expected behavior

After processing the message function app should stop consuming lots of CPU.

Actual behavior

Function's host continues consuming 100% CPU even after the function has nothing to do.

Known workarounds

N/A

Related information

• Programming language used: C#
• Bindings used: Storage Queue Trigger

The problem looks to be very similar to https://github.com/Azure/azure-functions-host/issues/4599, but in my case I have isolated mode which I'm unaware how to profile.

[bhagyshricompany]

Thanks for reporting will check and update.

[andriysavin]

@bhagyshricompany I stopped that function to free CPU for other services in that service plan. Let me know if you need me to start it to reproduce the issue.

[bhagyshricompany]

Hi @fabiocav please comment and validate.

[acorn20180214]

The reported problematic app had dynamic concurrency enabled, so same issue with https://github.com/Azure/azure-sdk-for-net/issues/40411, it's not a matter of whether isolated model or not, but it's a matter of queue trigger + dynamic concurrency, we can mitigate the issue by disabling dynamic concurrency. There is an ongoing internal ticket investigating it and will update it later as well.