Queue Trigger Function occupying memory and not releasing even in idle state in AKS POD

[dileept84]

Following the recent deployment of our Azure Queue Trigger Function within AKS, we’ve observed that memory consumption continues to increase even when the pod is in an idle state (i.e., not actively processing messages). Notably, the memory is not being released over time, leading to sustained high usage.

Upon analyzing memory dumps from the affected pods, we identified that the primary objects consuming memory are:

System.Threading.TimerQueue Microsoft.Azure.WebJobs.Utility+<DelayWithBackoffAsync>d__24

These objects are typically associated with internal polling mechanisms used by the Azure WebJobs SDK, particularly for managing background listeners and retry logic.

Below are the objects which we can see in memory dumps: AsyncStateMachineBox<VoidTaskResult,Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+<UnsynchronizedStartHostAsync>d__53> AsyncStateMachineBox<VoidTaskResult,Microsoft.Azure.WebJobs.Script.Utility+<DelayWithBackoffAsync>d__24> DelayPromiseWithCancellation TimerQueue

Currently we are disposing the unused memory after every 5 mins but the above objects are getting release. any help or suggestion is appreciated. Thank You!

[Frulfump]

Typo in title Ideal should be idle? And regarding

Currently we are disposing the unused memory after every 5 mins but the above objects are getting release.

Are they getting released or are they not getting release? If they are not it probably(I think) means they are rooted in something? And just for clarification it's referring to these objects right?

AsyncStateMachineBox<VoidTaskResult,Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__53> AsyncStateMachineBox<VoidTaskResult,Microsoft.Azure.WebJobs.Script.Utility+d__24> DelayPromiseWithCancellation TimerQueue

[sharadrumane]

They are not getting release at all, yes those are referring same objects

[jviau]

@dileept84 can you clarify your expectations here? What is an "idle" state? If the host is running, then queue listeners will be running. It is most likely these objects are responsible for listening for new messages on the queue.

[dileept84]

hi @jviau,

Idle state means - There is no load on the application application is in running state but there is no api call or anything going to application. it just only application processes are running.

I have a doubt that with in 1 container we are running 5-6 queue trigger functions is it possibly those are causing the issue?

[jviau]

Can you quantify what "high usage" is here? And if you have 5-6 queue trigger functions then I would definitely expect continuous background work as the app monitors your queues for new messages. But we need to first determine if what you are seeing is beyond that expectation or not.

[dileept84]

/bot not-stale

[microsoft-github-policy-service[bot]]

Thank you dileept84. This issue will not be automatically closed and a member of the team will review it soon.