Encountered an error (InternalServerError) from host runtime.

[niranjan2020]

Hi I have created azure function and it is running fine as expected. Later I created arm template for same azure function and i run as piepline in azure devops against same resource. I have given incremental mode in azure devops. When i deploy my arm template it works fine. Its deploying and changes are taking place. Only problem i am facing at the end it throws below error

##[error]undefined: Encountered an error (InternalServerError) from host runtime.

Below is my template

{
            "type": "Microsoft.Web/sites",
            "apiVersion": "2018-11-01",
            "name": "[variables('fetchSciHubProductURLName')]",
            "location": "[parameters('location')]",
            "tags": {
                "BU": "[parameters('Division')]",
                "Environment": "[parameters('environment')]"
            },
            "kind": "functionapp",
            "properties": {
                "serverFarmId": "[parameters('serverfarms_APSERDEVDVLGENSEAWE01_Linux_externalid')]",
                "clientAffinityEnabled": false,
                "httpsOnly": false,
                "siteConfig": {
                    "reservedInstanceCount": "1",
                    "appSettings": [
                      {
                        "name": "AzureWebJobsStorage",
                        "value": "storage account"
                      },
                      {
                        "name": "APPINSIGHTS_INSTRUMENTATIONKEY",
                        "value": "[parameters('storageAccount01APPINSIGHTS_INSTRUMENTATIONKEY')]"
                      },
                      {
                        "name": "APPLICATIONINSIGHTS_CONNECTION_STRING",
                        "value": "[parameters('storageAccount01APPLICATIONINSIGHTS_CONNECTION_STRING')]"
                      },
                      {
                        "name": "FUNCTIONS_EXTENSION_VERSION",
                        "value": "~3"
                      },
                      {
                        "name": "FUNCTIONS_WORKER_RUNTIME",
                        "value": "dotnet"
                      },
                      {
                        "name": "WEBSITE_ENABLE_SYNC_UPDATE_SITE",
                        "value": "true"
                      },
                      {
                        "name": "WEBSITE_RUN_FROM_PACKAGE",
                        "value": "1"
                      },
                      {
                        "name": "WEBSITES_ENABLE_APP_SERVICE_STORAGE",
                        "value": "true"
                      },
                      {
                        "name": "Test",
                        "value": "test"
                      }
                    ]
                  }
            }
        },
        {
            "type": "Microsoft.Web/sites/functions",
            "apiVersion": "2018-11-01",
            "name": "[concat(variables('fetchSciHubProductURLName'), '/getproductsfromcoordinates')]",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[resourceId('Microsoft.Web/sites', variables('fetchSciHubProductURLName'))]"
            ]
        },
        {
            "type": "Microsoft.Web/sites/functions",
            "apiVersion": "2018-11-01",
            "name": "[concat(variables('fetchSciHubProductURLName'), '/UploadFilesToAzureStorage')]",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[resourceId('Microsoft.Web/sites', variables('fetchSciHubProductURLName'))]"
            ]
        }

can someone tell me why this issue is coming? Thank you

[KenSpur]

Same here, cant seem to add a predefined empty function with Microsoft.Web/sites/functions.

Or run the template when the function already exists.

My use case would be to create a function app with a predefined function and key, to then register it as backend in api managment and use the function specific key in the header.

[amansahni1]

Hi,

Was someone able to resolve this. Getting the same error.

I am using the Terraform data function azurerm_function_app_host_keys to retrieve the host keys and getting the below... Error: Error making Read request on AzureRM Function App Hostkeys "XXXXXX": web.AppsClient#ListHostKeys: Failure responding to request: StatusCode=400 – Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="Encountered an error (InternalServerError) from host runtime." Details=[{"Message":"Encountered an error (InternalServerError) from host runtime."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","Message":"Encountered an error (InternalServerError) from host runtime."}}]

[Jay-Madden]

bump anyone else seen this?

[v-bbalaiagar]

Hi @balag0, Could you please look into this issue

[balag0]

Could someone provide more details on the empty predefined function. Is the app using a valid zip file with host.json but no functions? Or is the app using WEBSITE_RUN_FROM_PACKAGE = 1 but with no content deployed? The app name would also be very helpful. Thanks

[DataGobes]

I am joining the club. So for me I get this error when I run an ARM template with a listkeys like @amansahni1 against a function app which does not have a zip deployed yet.

"[listKeys(concat(resourceId('Microsoft.Web/sites', variables('functionAppName')),'/host/default/'), '2019-08-01').functionKeys.default]"

I was running this ARM template a couple of times succesfully when I had a python ZIP deployed, but when I deleted the app and completely redeployed the deployment started to fail with:

undefined: Encountered an error (InternalServerError) from host runtime

In the same template I also request the function hostname, and that one is no problem because it is stored in my vault after the failed deployment. Only thing missing is the function app key.

{ "comments": "Add the Function key in the Key Vault", "type": "Microsoft.KeyVault/vaults/secrets", "name": "[concat(variables('keyVaultName'), '/rapidfunctionkey')]", "apiVersion": "2018-02-14", "location": "[parameters('location')]", "dependsOn": [ "[resourceId('Microsoft.Web/sites', variables('functionAppName'))]" ], "properties": { "value": "[listKeys(concat(resourceId('Microsoft.Web/sites', variables('functionAppName')),'/host/default/'), '2019-08-01').functionKeys.default]" } }

[razeenh]

My arm template was working fine before, until last week when I discovered that the host keys for the function app was no longer being set despite the deployment succeeding. It does seem to get created when creating manually via Azure Portal though, so have no idea why the arm template deployment is resulting in this issue. Its created against an existing general purpose V2 storage account and S1 app service plan.

I also see the same InternalServerError error when attempting to execute the function, which is likely due to the absence of the host key.

Is there a list of troubleshooting steps when it comes to the deployment of a function app?

[vincenttermaat]

Any insights? Facing the same issue.

[v-bbalaiagar]

Hi @balag0, Kindly can you look at this issue here

[Jay-Madden]

So we had the error with terraform deployments and we solved it like this https://github.com/terraform-providers/terraform-provider-azurerm/issues/11266#issuecomment-818757226

I dont know how helpful this solution will be for ARM solutions but we had the same InternalServerError

[DataGobes]

Ah that is definitely worth a try, looks like te exact same issue. Thanks for mentioning this.

[v-bbalaiagar]

Can we know if the above mentioned comment resolves your query, or if you have any further queries.

[RobBowman]

I've seen this error a few times lately when running my bicep template to deploy a function app. It seems to be intermittent - without any changes, a subsequent run of the same template works ok.

[NicoleMyM]

Try this, because in my case I can after many tries

{ "name": "FUNCTIONS_WORKER_RUNTIME", "value": "dotnet" }, { "name": "WEBSITE_ENABLE_SYNC_UPDATE_SITE", "value": "true" }, { "name": "WEBSITE_RUN_FROM_PACKAGE", ----> Remove that "value": "1" }, { "name": "WEBSITES_ENABLE_APP_SERVICE_STORAGE", "value": "true" }, { "name": "Test", "value": "test" } My appsetting

[ajkimberley]

Has there been any progress on this? I'm getting the same error for multiple Functions each time I try to deploy. Note, it's a different set of functions that throw this error each time I deploy, with no apparent pattern to which ones will fail.

[NicoleMyM]

Has there been any progress on this? I'm getting the same error for multiple Functions each time I try to deploy. Note, it's a different set of functions that throw this error each time I deploy, with no apparent pattern to which ones will fail.

I could, in the previous comment I left my appsetting. Do not define WEBSITE_RUN_FROM_PACKAGE

{ "name": "WEBSITE_RUN_FROM_PACKAGE", ----> Remove that "value": "1" },

[ajkimberley]

Many thanks for the response. Unfortunately, that didn't make any difference for me.

I have noticed that for me, it only fails when I have > 15 functions defined in the Bicep file. Though when it errors it's indeterminate which functions that error occurs with. I.e., it's a different set of functions that fail to deploy each time I try. Very confusing.

[vincenttermaat]

Hm. I must have solved this at some point. A year ago. I did NOT use Bicep by the way, so I guess that's unrelated. Can you start with 1 and move up by 5 each time? What is the error again? Does the resource group activity log provide more details? (json response)

Wild guess: is the deployment executed paralel or in series? If it fails at a different one each time, perhaps that could be a reason.

[ajkimberley]

@vtermaat Thanks for the response and for the questions. Really appreciate it.

I tried compiling the Bicep to an ARM template and deploying that way, but that made no difference so I don't think it's related to it being a Bicep file.

By "starting with 1 and moving up by 5 each time" do you mean declaring each group of functions in separate Bicep files? Or is there some way in a Bicep file/ARM template to declare the how you want the deployment to be batched? I'm fairly new to Bicep (and have never used raw ARM) so I could be wrong, but I would guess that given the declarative nature of Bicep, there would be no functionality to specify how you want the resources deployed.

The error is:

{
    "status": "Failed",
    "error": {
        "code": "BadRequest",
        "message": "Encountered an error (InternalServerError) from host runtime.",
        "details": [
            {
                "message": "Encountered an error (InternalServerError) from host runtime."
            },
            {
                "code": "BadRequest"
            },
            {}
        ]
    }
}

This occurs for each function that fails to deploy, but which specific functions fail (and how many) appears to be indeterminate.

Oddly, the functions that fail actually do appear in the Function App resource, when I inspect in the Portal. But the problem is, the resources that depend on those functions (in this case, my Event Grid Subs) do not get deployed.

I'm not sure how the deployment is executed. Is there a way of telling? I just run the bicep file using:

az deployment group create --resource-group <resource-group-name> --template-file <path-to-bicep>

It definitely deploys the dependencies of the functions and function app first though. And the Functions don't depend on each other so it shouldn't make a difference which order they get deployed in.

Thanks again for your help!

[vincenttermaat]

Is this the (only) deployment error that shows up in the activity log? This activity log also shows you what went successfully and in what order: going into "how is the deploy executed".

[ajkimberley]

@vtermaat Ah apologies, I was looking at the Deployments page instead of the Activity log. The activity log gives me a bit more detail. I attempted a redeploy earlier and here are some screenshots (from top-to-bottom) of the activity log for that redeployment:

Here the JSON for one of the Update Web Apps Functions:

{
    "authorization": { ... },
    "caller":  ... ,
    "channels": "Operation",
    "claims": { ... },
    "correlationId": "c4523426-6fd4-4daf-b7df-4234eaa3a80d",
    "description": "",
    "eventDataId": "0cc45d55-9c6f-4e28-8027-f04abd6087d4",
    "eventName": {
        "value": "EndRequest",
        "localizedValue": "End request"
    },
    "category": {
        "value": "Administrative",
        "localizedValue": "Administrative"
    },
    "eventTimestamp": "2022-05-05T08:02:49.6378813Z",
    "id": "/subscriptions/7209fdce-9fd9-4067-b65a-88f2f51946d5/resourcegroups/TestStripeDeployment/providers/Microsoft.Web/sites/adam-stripe-functions/functions/GetProductPrices/events/0cc45d55-9c6f-4e28-8027-f04abd6087d4/ticks/637873345696378813",
    "level": "Error",
    "operationId": "cc97c523-8d15-4129-85ea-3a1d757a845c",
    "operationName": {
        "value": "Microsoft.Web/sites/functions/write",
        "localizedValue": "Update Web Apps Functions"
    },
    "resourceGroupName": "TestStripeDeployment",
    "resourceProviderName": {
        "value": "Microsoft.Web",
        "localizedValue": "Azure Web Sites"
    },
    "resourceType": {
        "value": "Microsoft.Web/sites/functions",
        "localizedValue": "Microsoft.Web/sites/functions"
    },
    "resourceId": "/subscriptions/7209fdce-9fd9-4067-b65a-88f2f51946d5/resourcegroups/TestStripeDeployment/providers/Microsoft.Web/sites/adam-stripe-functions/functions/GetProductPrices",
    "status": {
        "value": "Failed",
        "localizedValue": "Failed"
    },
    "subStatus": {
        "value": "BadRequest",
        "localizedValue": "Bad Request (HTTP Status Code: 400)"
    },
    "submissionTimestamp": "2022-05-05T08:03:58.1408502Z",
    "subscriptionId": "7209fdce-9fd9-4067-b65a-88f2f51946d5",
    "tenantId": "0ab58345-e2f1-4b6c-a4e8-2324022df52d",
    "properties": {
        "statusCode": "BadRequest",
        "serviceRequestId": null,
        "statusMessage": "{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (InternalServerError) from host runtime.\",\"Target\":null,\"Details\":[{\"Message\":\"Encountered an error (InternalServerError) from host runtime.\"},{\"Code\":\"BadRequest\"},{\"ErrorEntity\":{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (InternalServerError) from host runtime.\"}}],\"Innererror\":null}",
        "eventCategory": "Administrative",
        "entity": "/subscriptions/7209fdce-9fd9-4067-b65a-88f2f51946d5/resourcegroups/TestStripeDeployment/providers/Microsoft.Web/sites/adam-stripe-functions/functions/GetProductPrices",
        "message": "Microsoft.Web/sites/functions/write",
        "hierarchy": "7209fdce-9fd9-4067-b65a-88f2f51946d5"
    },
    "relatedEvents": []
}

[ajkimberley]

I don't know if this is some odd implementation detail or part of the bug, but it looks like each step is actually being duplicated. E.g., Create Deployment is listed twice, each Update Web App Function is listed twice.

[vincenttermaat]

I personally can't go into this right now, maybe someone else can, but this log and it's errors should give you a better grasp of what's going on.

[ajkimberley]

@vtermaat no problem. Thanks for the pointers.

[RDavis3000]

Same issue here! Heres a snippet from the Activity log. Bad Gateway on the function app? I'm running my template from the portal or using the azureDeploy.ps1 in Visual Studio, dont know if it would work better using Azure Pipelines

"correlationId": "6260ce8a-1af5-4b91-8d7a-9762026ac538",
"description": "",
"eventDataId": "fa494796-c37f-456f-b5a5-ceba673a3e1d",
"eventName": {
    "value": "EndRequest",
    "localizedValue": "End request"
},
"category": {
    "value": "Administrative",
    "localizedValue": "Administrative"
},
"eventTimestamp": "2022-05-11T09:33:37.600103Z",
"id": "/subscriptions/814f37ac-6bb0-4108-9c29-72909f9eef1e/resourcegroups/rg-d-lxr-aue-ingestion/providers/Microsoft.Web/sites/fa-d-lxr-aue-ingestion/functions/ITwoCXDocSettingsMapDocSettingsToStatusActivity/events/fa494796-c37f-456f-b5a5-ceba673a3e1d/ticks/637878584176001030",
"level": "Error",
"operationId": "3eb52638-a79c-48fd-b888-219b7ed0cdb8",
"operationName": {
    "value": "Microsoft.Web/sites/functions/write",
    "localizedValue": "Update Web Apps Functions"
},

[probertdaniel]

I have this same issue (June 2022) - I'm in the same scenario as OP, ARM Template deploying a Function App Service. There are no functions deployed (they get deployed after this ARM Deployment). Using DevOps for deployment. I'm not explicitly doing anything with Function Keys in the ARM template. If I run the deployment again and again, it eventually succeeds, i.e. this is an intermittent error. And I have the same errors as seen in the Activity Log:

I'll try removing "WEBSITE_RUN_FROM_PACKAGE" from config and see if that changes anything. Also, I recently upgraded this service to v4 runtime (.NET 6) and the issues seem to have started since then. The App Service is already deployed i.e. it's not deploying it for the first time, this is an incremental deployment.

[Saj1234]

I was able to work around this issue by deleting the function app and recreating it. Probably not the most practical approach for a workaround.

Here is my scenario: I was getting the same issue "Encountered an error (InternalServerError) from host runtime.* when I read the function keys using listkeys method in bicep template.

This is where it failed in my template:

listkeys('${functionApp.id}/host/default', '2016-08-01').functionKeys.default

The deployment fails at this point giving the error:

Encountered an error (InternalServerError) from host runtime.

So, it turns out that the function deployment has not created the _master and default function keys during the function deployment.

You can use the bicep templates to create a custom key if needed and read that. However, for me, it created the keys on the next run after making the changes below. Still might be an intermittent thing.

Anyway, re-ran the pipeline after deleting the function app created from the previous run and it created the function and the keys this time.

I hope this will be helpful to someone!

[captainjono]

have this same issue (June 2022) - I'm in the same scenario as OP, ARM Template deploying a Function App Service. There are no functions deployed (they get deployed after this ARM Deployment). Using DevOps for deployment. I'm not explicitly doing anything with Function Keys in the ARM template. If I run the deployment again and again, it eventually succeeds, i.e. this is an intermittent error.

My functions get created without master keys :(

This is still happening today! If your a newbie to functions, its actually quite hellish debugging this, you will scratch your head and wonder why every guide you follow fails deploy successfully!!

[ronhowe]

I came across this issue this week at work and found this thread.

+1 for the irritation of it all.

[dinakar10m]

Any updates on this issue? I am using terraform, works fine for a couple of iterations then I start seeing this error. I am using azurerm_function_app_host_keys data source to retrieve the keys but the underlying error is coming from azurerm provider. I've tried everything suggested on GitHub - removed WEBSITE_RUN_FROM_PACKAGE, added container name containing the keys explicitly in the app settings no luck so far.