Azure functions Http triggers getting down and throws 503/502 and sometimes even requests are not reaching to Http trigger

[SainiAshish]

We have a azure function app(App service plan) in which multiple http triggers are present. Sometimes functions app gets down and started throwing 503/502 and after restarting 2-3 times it starts working. Sometimes even request are not reaching to the http trigger and request gets stuck.

Is there any way to find the root cause of the issue or any telemetry why it is happeing.

[ConnorMcMahon]

I would recommend going to the portal and navigating to Platform Features > Resource managent > Diagnose and solve problems. We have various detectors and troubleshooting guides that help diagnose the most common problems people have with Azure Functions.

If that doesn't help you solve your problem, provide us with timestamps, and share you function application name, and we can diagnose the issue from our end. If you don't feel comfortable sharing your function app name publicly, you can share it privately.

[EmilAlipiev]

Hi did you find the reason for this? i have had several 503 reports in my appcenter logs but in Appinsights i cannot find anything related to this. so appcenter logs shows only for clientside mobile app. that sounds like that requests arent even reaching to server. When I check Application insights neither under failures nor under failed requests, i have no single entry. I am not sure how to use Diagnose and solve problems, i tried everything but it shows all healthy. why client is reporting 503?

Ps, my azure function app is linux, not windows. do you also have linux or windows based?

[balag0]

If you can share the appname we can investigate. https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately

[EmilAlipiev]

just a quick question i had 503 for a function it seems to be an exceeded timeout issue i found somewhere in logs and right after it several 404 not found exceptions although urls are correct. what is actually happening? is the azure function app restarting or going down?

[balag0]

It could be due to the worker restarting. But we can only guess without the appname :)

[EmilAlipiev]

how to report it privately? when i use support option in azure portal, it always redirect me some general documentation. my functions app is throwing 503 server not available sometimes and app insights shows nothing about it. No logs available. Before it was much easier to contact support, now looks like only redirecting to some general documentation

[balag0]

You can follow these steps to report the app name privately. https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately

[RizwanSharp]

@balag0 Hello Team, Our app started getting 503s followed by 404 (on valid URLs as someone reported above). There was a bug in the code which caused a sort of infinite loop and caused thousands of errors. Now that loops has been exited but we are in a state where we are getting 503. Can you please help us understand the cause and fix so we can avoid this situation without requiring a restart of the function app?

Our function app is: wb-qa-southcentralus-orderprocessing-fn.azurewebsites.net.

Any help would be greatly appreciated.

[RizwanSharp]

We did notice, we were running too many apps on a couple of S1 servers and we have scaled those up to P1 (2).

[balag0]

@RizwanSharp Took a quick look and found a few storage exceptions but looks like they might have been transient. If you see similar errors again, please open a new issue and we can investigate further.

[RizwanSharp]

Appreciate for taking a look, yes we did have some exceptions + running many apps on limited resources. Scaling up took care of the problem. Thanks again!

[mikecole]

I'm having the same issues. It's a mix of 404, 503, and a few others. Very strange. It's sporadic.

I believe this is what you need, if not let me know: 2021-06-07T15:57:43.315 [Information] Executed 'detail' (Succeeded, Id=e0e4fc12-9664-4a86-a084-4fbfbafd9763, Duration=28ms)

[eaquiler]

Hi @balag0, we are also experiencing a similar issue, the function was working fine until Jun 14th (or at least that's what we can find in the AI). There are no error messages when the function is run from the Azure site but only HTTP 503, when called from outside Azure it gets 502 (because the internal 503). There are several log messages in the function none of them are shown.

2021-06-16T10:42:55.737 21b3a8de-b067-4734-98e5-d9fd0b893d1a North Europe

Thanks in advance for any help

[balag0]

@eaquiler Looks like the errors were due to an issue with the recent functions runtime update (3.0.15960) That update has now been rolled back and I can see this app is now running on 3.0.15733, so there shouldn't be any more failures.

[eaquiler]

@balag0 sorry it seems that I accidentally deleted my previous message. I said I verified that it is running on 3.0.15733 but the issue still there, I see that the process seems to be restarting on the middle of the request as shown below

Connected!
2021-06-16T17:14:09Z   [Information]   Executing 'Functions.Http***' (Reason='This function was programmatically called via the host APIs.', Id=5ae40822-6a3c-4808-9dfa-91b931906b91)
2021-06-16T17:14:09Z   [Verbose]   FunctionDispatcher state: WorkerProcessRestarting

Thanks in advance for any guide or help!

[balag0]

Looks like this was due to an attempt to create a new function named '****Search' That operation failed with below exception which led to the restart

System.UnauthorizedAccessException : Access to the path 'C:\home\site\wwwroot****Search\function.json' is denied.

In general write operations will always lead to workers recycling.

[eaquiler]

We opened a case because we couldn't find the reason for the Access denied error and we discover that the node version we were using in the settings are deprecated and not available anymore in the server (very hard to find it out from an Access denied message). After updating the node version to the current newest (~14) it start to work again. So, if someone get this error in the future just check node version in addition to the runtime.

[wernerchao]

Hi, I am experiencing the same issue. My function has run seamlessly until very recently that I start to get transient 502 error. And upon restarting the app it would then work fine again for some time, then the error would come back again later. This keeps repeating.

• Here is my function detail

UTC execution time: 2022-06-20 21:01:21.995
execution Id: b03278c8-8e5b-4fc4-bc06-e987f0e523c2
Region of Function App: Central US

• Here is the transient error I got

{
    "statusCode": 502,
    "headers": {
        "Date": "Mon, 20 Jun 2022 18:46:17 GMT",
        "ETag": "\"62686d80-f9\"",
        "Server": "nginx",
        "Content-Length": "249",
        "Content-Type": "text/html"
    },
    "body": "<html>\n    <head>\n        <title> Server Error </title>\n    </head>\n    <body>\n        <font color =\\\"#aa0000\\\">\n            <h2>Server Error.</h2>\n        </font>\n\n        There was an unexpected error in the request processing.\n    </body>\n</html>"
}

Please provide some work around, maybe switching region or something. Because this did not happen in the past, and the frequency of error is increasing. Thanks!