Random "The service is unavailable." and "Azure Functions runtime is unreachable" errors

[Arjan321]

We have been running a couple of Azure Functions in various subscriptions, and every once in a while (about once a week), the entire Azure Function goes down and reports "The service is unavailable." when accessing the Function-app via HTTP and reports "Azure Functions runtime is unreachable" in the Azure Portal.

HTTP response:

Portal:

The issue appears randomly, without any changes in our end (no deployment, etc.) and also resolves randomly without any interaction from our side.

In the "Activity log" an error is written for the job "Sync Web Apps Function Triggers" with status "Failed":

        "statusCode": "BadRequest",
        "statusMessage": "{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (ServiceUnavailable) from host runtime.\",\"Target\":null,\"Details\":[{\"Message\":\"Encountered an error (ServiceUnavailable) from host runtime.\"},{\"Code\":\"BadRequest\"},{\"ErrorEntity\":{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (ServiceUnavailable) from host runtime.\"}}],\"Innererror\":null}",

The issue seems quite similar to #8519, however we are running Linux.

This is causing quite some problems, since we are no longer able to provide reliable service to our end-users.

Investigative information

• Timestamp: Wednesday, 27 July 2022, 07:47, Coordinated Universal Time (UTC)
• Function App version: V4
• Function App name: Please use Invocation ID
• Function name(s) (as appropriate): All functions within the app are non-working
• Invocation ID: None of the functions work, but to find our function-app: 2022-07-27T08:05:13.773Z - e95ba16f-f2a2-420c-bcfd-4891d54198ae
• Region: West-Europe

Repro steps

None that we can find

Expected behavior

Always work

Known workarounds

None

Related information

Hosting Model: Consumption Plan OS: Linux Version: V4 Hosting Model In-Process Language: C#/dotnet6 Configuration:

• WEBSITE_RUN_FROM_PACKAGE points to a valid file on a storage account
• WEBSITE_MOUNT_ENABLED: 1
• WEBSITE_ENABLE_SYNC_UPDATE_SITE: false
• WEBSITE_CONTENTAZUREFILECONNECTIONSTRING: Valid keyvault reference toa a storage account
• SCM_DO_BUILD_DURING_DEPLOYMENT: false
• FUNCTIONS_WORKER_RUNTIME: dotnet-isolated
• FUNCTIONS_EXTENSION_VERSION: ~4
• AzureWebJobsDisableHomepage: true

[wbail]

Hi Arjan32,

Normally this error message is related with misconfiguration.

Please take a look the bullets below:

• Add the variable AzureWebJobsStorage and add the value of the connection string from the Storage Account Reference:

  • https://docs.microsoft.com/en-us/azure/azure-functions/functions-app-settings#azurewebjobsstorage

• Double check the value of the variable WEBSITE_RUN_FROM_PACKAGE Reference:

  • https://docs.microsoft.com/en-us/azure/azure-functions/run-functions-from-deployment-package
  • https://github.com/projectkudu/kudu/wiki/WEBSITE_RUN_FROM_PACKAGE-and-WEBSITE_CONTENTAZUREFILECONNECTIONSTRING-Best-Practices

[Arjan321]

As mentioned in then original issue, Those values point to valid values.

Given the extreme randomness of both the issue appearing and resolving itself, I hardly doubt any setting on our end is responsible.

[gilesmatthews]

Hi, is there any update on this issue? I am seeing the same random service unavailable issues. Thanks

[rlucassen]

Same issue here, completely random. Waiting on a fix for a while now.

[LuckyLub]

Sorry for you guys, but I'm happy to read this... thought I was going crazy. What region do you guys have your Azure Functions running? Ours are in West Europe.

[rlucassen]

Sorry for you guys, but I'm happy to read this... thought I was going crazy. What region do you guys have your Azure Functions running? Ours are in West Europe.

Mine are running in West Europe as well, same for @Arjan321

[tomabg]

same region West Europe

[tomabg]

this also breaks terraform deployment...as we are on test only we will try different region soon

[rlucassen]

this also breaks terraform deployment...as we are on test only we will try different region soon

Curious to know if other regions do work.

Heard that Microsoft is doing updates in the West Europe region around next week

[LuckyLub]

We also deploy with Terraform BTW. Changing the location to Central US seems to do the trick.

[jnekrasov]

Any news on this?! We are experiencing the same problems in West Europe region

[Ralle1986]

Also seeing issue in West Europe region

[balag0]

Sorry for the delay in responding. Yes, there was a regression due to a recent update in the region which caused intermittent errors. The fix rollout has already started and in progress currently. If there are any apps still experiencing failures, could you please share the details and we can double check them. Thanks

[LuckyLub]

Before we move back, can you please confirm when the roll-out is completed?

[lightwaver]

any news on that topic or link to the issue to see if its solved ?

[LuckyLub]

@balag0 any update?

[LuckyLub]

@surgupta-msft, @balag0, any updates? Just tried to redeploy to West-Europe, still running into random "The service is unavailable." errors Running the same functions in North-Europe works just fine.

[LuckyLub]

Just had contact with Azure's Help + Support. I presented the problem, it seemed to be known. However, they will collect some logs from my Azure Functions and look further into it. They are still working on upgrading the services in West-Europe. Currently it was advised to use another region.

[jeroenvermunt]

I can confirm that it is still occurring

[Rutix]

We have had these problems too. We have been in contact with Azure Support and they are saying the following:

" Conclusion: The 503s were detected ONLY on Azure Front End, the Front End instances encountered some unexpected error at that moment and weren’t able to handle the http requests and distributed them to specific workers that were hosting your function app.

.....

Let’s take a closer look at these 2 Front End instances at that time, Front End instance 24 encountered an error when trying to get a worker from the data role, and the same situation for the instance 5.

Unfortunately, this is an underlying platform issue as the Front End is an important component inside of Azure platform, and both the user side and our side cannot have action to interact with it, we apologize for all the inconvenience caused, but please rest assured that I’ve already reflected this to the Microsoft Azure Team and they already know this, we met this kind of issue before. "

^ this has been now several months ago. The fact that these problems are still popping up is disappointing. Also telling us to a different region is also disappointing. We are bound by compliance issues so we cant leave our region as easy.

[rlucassen]

Last wednesday we got the message from Microsoft that this issue should be fixed, we already transferred to a premium plan because we got tired of it after 4 months. I'm curious to know if anybody is still encountering this issue in the West-Europe region?

[LuckyLub]

So using premium is a valid work around?

[rlucassen]

So using premium is a valid work around?

Yes upgrading to premium worked for us

[Arjan321]

So using premium is a valid work around?

If "Throwing money at the problem" counts as a workaround, then yes. This ticket is specifically about the Consumption Plan.

[LuckyLub]

Just good to know what options are out there.

[LuckyLub]

Message from MS:

Product Group is still working on improvement on the west Europe region. Will keep you updated on any progress.

[Rutix]

I got some messages that we got hit again. We are not entirely sure if it was the same cause as this issue but we had "Service unavailable" a couple of times today in west europe.

[rdvansloten]

@Rutix I happened upon this thread as well this afternoon, also from The Netherlands, having Functions (Consumption tier) in West Europe. They were unavailable and/or throwing SSL errors. I also could not deploy from VS Code (unavailable)

What fixed it for me is going into the Portal and Restarting my Functions manually.

[LuckyLub]

That’s a temporary fix.

[Rutix]

@rdvansloten sadly that is only a temp fix and also doesn't always work (or takes a long time to recover). The best way would be for the Azure team to fix this asap.