[Premium] Function App errors when storage account has VNET ACL

[ishepherd]

We want to add a VNET ACL to the storage account used by a function app

The networking page says this can be done, if you can wait 12 hours: https://docs.microsoft.com/en-us/azure/azure-functions/functions-networking-options#restricting-your-storage-account-to-a-virtual-network,

We've found it doesn't work. Gives access denied trying to access D:\home\LogFiles\Application\Functions\Host

Actually, the page I linked to above explains this error:

There are some things that VNet Integration doesn't support including: Mounting a drive

Can this be fixed? We have a requirement to lock down all resources.

[ishepherd]

paraphrasing off-thread with @alexkarcher-msft: This is supposed to work - if left alone long enough. The documentation point "doesn't support ... Mounting a drive" is unrelated.

I will see if I can repro the issue.

[ishepherd]

@alexkarcher-msft I have a repro for this now. UnauthorizedAccessExceptions for 16 hours and counting.

Access to the path 'D:\home\LogFiles\Application\Functions\Host' is denied. 

System.UnauthorizedAccessException:
   at System.IO.FileSystem.CreateDirectory (System.IO.FileSystem, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.IO.Directory.CreateDirectory (System.IO.FileSystem, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.IO.Abstractions.DirectoryWrapper.CreateDirectory (System.IO.Abstractions, Version=2.1.0.227, Culture=neutral, PublicKeyToken=96bf224d23c43e59)
   at Microsoft.Azure.WebJobs.Script.FileUtility.EnsureDirectoryExists (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Extensions\FileUtility.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 38)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.InitializeFileSystem (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 433)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.PreInitialize (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 425)
   at Microsoft.Azure.WebJobs.Script.ScriptHost+<InitializeAsync>d__80.MoveNext (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 256)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.ScriptHost+<StartAsyncCore>d__79.MoveNext (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 237)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Extensions.Hosting.Internal.Host+<StartAsync>d__9.MoveNext (Microsoft.Extensions.Hosting, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+<UnsynchronizedStartHostAsync>d__37.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebJobsScriptHostService.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 209)

[ishepherd]

Edit Restarting the app did not help.

[ishepherd]

Off-thread with @alexkarcher-msft: The issue is understood and a fix is coming

Alex you might consider updating the relevant doc in the meantime - Cheers

[alexkarcher-msft]

Good call out! I have a PR out to update the docs

https://github.com/MicrosoftDocs/azure-docs-pr/pull/92236

[ishepherd]

@fabiocav you closed this, but @alexkarcher-msft is the fix still coming? How will we know when it's arrived?

[alexkarcher-msft]

Ah, sorry. I confused this with a separate issue. We identified a workaround for this scenario, and need to fully document it.

It looks like you can connect to a secured storage account using run from package as a URL and that will allow your code to be stored in a VNET secured storage account https://docs.microsoft.com/en-us/azure/azure-functions/run-functions-from-deployment-package#enabling-functions-to-run-from-a-package

With that configuration you will need a separate storage account with just your code as a zip. Then, you should be able to secure your other account with VNET service endpoints.

I haven't had a chance to test this configuration, but this is the current workaround we're tracking :)

[ishepherd]

Thanks @alexkarcher-msft

Good that there's a workaround but anything that compromises reliability is a no-go - Below re an incident we had on 2019-05-28, investigated by @FinVamp1. our ref SMDS-1894. Also your link notes the impact on cold start performance which is not great either.

Will we be able to use WEBSITE_RUN_FROM_PACKAGE = 1 and any guesses on ETA?

---

For the 404 issue on the 28th of May three of the four workers assigned to the site ran into this pattern.

1. The worker is assigned.
2. It goes to download the Zip file from Storage for the application.
3. The Storage Account takes time to respond (up to 29 seconds on two of the instances.)
4. The Zip download fails.
5. A file called "FAILED TO DOWNLOAD ZIP FILE.txt" is created at D:\Home\Site\wwwroot
6. As we have no Host.json we create an empty one.
7. We cannot save this to disk as D:\Home\Site\wwwrot is still read only.
8. We send requests to this worker anyway and these return a 404.

[deyanp]

@alexkarcher-msft , is it still not possible to secure the underlying storage account of a (Premium Plan) Function App with Vnet integration? How should I undrestand "wait 12h" - we cannot have a downtime for 12h if func app cannot connect to storage account ... Is WEBSITE_RUN_FROM_PACKAGE = 1 the only way how this can work? I understand it is not recommended due to cold start issues, or is it fine for Premium Plan Function Apps?

[jeffhollan]

At this moment you shouldn’t need to wait for the VNet routes to establish when adding a service endpoint / private link service to call from your function. However, you still need some storage account that doesn’t have a VNet for the WEBSITE_CONTENTFILECONNECTIONSTRING app setting. That’s where your code is stored when doing general publish gestures (anything but run from package = some URL to a zip file). The feature to allow even that storage account to be behind a VNet via private link is checked in and rolling out in our next platform update. Likely about 2 months out.

[deyanp]

@jeffhollan this is great news, assuming I got it correctly - we will be finally able to network/firewall-secure (using service endpoints and vnet integration) the underlying storage account of a function app in 2 months!

Or will only private link be supported, and no service endpoints?

[alexandrelimait]

Any updates? service endpoint for storage accounts and KeyVault really impact many customers.

[justinimel]

Same Issue

[Carrie810]

I got the same issue

[jeffhollan]

Still coming - best ETA would be end of this year. We hit an issue with the deployment for the earlier ETA so doing a bit phased rollout later this year

[alona-shevliakova]

same issue

[fanasere]

same issue

[jeffhollan]

Heads up preview just went live for West Europe - let us know if you have any issues with this and hope to push it out globally in the coming weeks https://docs.microsoft.com/en-us/azure/azure-functions/functions-networking-options#restrict-your-storage-account-to-a-virtual-network-preview

[JamesDLD]

Same issue here, will follow this issue.

[VishalNagarAR]

Thanks @jeffhollan for mentioning that link. I have my solution for West Europe and it did work. Following changes were done in function appsettings to make it work.

WEBSITE_CONTENTOVERVNET = 1. WEBSITE_VNET_ROUTE_ALL = 1. WEBSITE_DNS_SERVER =168.63.129.16.

[knaidoo33]

I have this issue as well. @jeffhollan, will the preview be available globally soon? I am in West US 2 and would like to try it.

[jeffhollan]

Just updating - this is now GLOBAL, and generally available.

[Bunzab]

Just updating - this is now GLOBAL, and generally available.

Any ETA on when this will support Linux Plans please?

[blueelvis]

@jeffhollan - I am still not able to get this working with a storage account secured by Private Endpoint. I did create a PV3 App Service plan (To ensure we are on the latest scale unit) and then scaled it to EP1 using ARM Template but I still keep on getting -

Access to the path 'C:\home\site\wwwroot' is denied.
System.Private.CoreLib: Access to the path 'C:\home\site\wwwroot\host.json' is denied.

I have the following App Settings in place -

- AzureWebJobsStorage
- WEBSITE_CONTENTAZUREFILECONNECTIONSTRING
- WEBSITE_CONTENTOVERVNET = 1
- WEBSITE_VNET_ROUTE_ALL = 1

Both AzureWebJobsStorage & WEBSITE_CONTENTAZUREFILECONNECTIONSTRING are backed by a Storage Account which is accessible only over a Private Endpoint. Here is the support ticket # for reference - 2102280040000493

Thanks, Pranav

[DeepMalh44]

@blueelvis I am facing the same issue and I am not sure if your support ticket got any resolution? Can you please share if there was any resolution. I am facing exactly same issue as you are having and its now July 2021 and I have ensured the configuration have all those App settings as well. @jeffhollan any help on this please?

[DeepMalh44]

Resolved: it seems that adding the "WEBSITE_RUN_FROM_PACKAGE" = 1 app settings to the Function App settings gets rid of that error.

[judedaryl]

@ketaanhshah did you run into a 503 error on the scm? i get service unavailable the function app gets created

[victormendozalex]

@blueelvis I am facing the same issue and I am not sure if your support ticket got any resolution? Can you please share if there was any resolution. I am facing exactly same issue as you are having and its now July 2021 and I have ensured the configuration have all those App settings as well. @jeffhollan any help on this please?

This helped me.

[ben-002]

We are facing the same error in a function with a dedicated App service plan in production.

We have a storage account that we secured and had no problems in the staging environments. Our staging environments use a dedicated app service plan as well, but is not a premium: tier - Standard, size -S1 , capacity - 1. However, the above mentioned error appeared in production: tier - PremiumV2, size -P1v2 , capacity - 2.

We had to revert the security changes in prod and now I'm trying to reproduce the problem in staging but it seems to be really tricky. I have not managed to reproduce the error even after scaling up the staging environment to the same tier as in prod. Does anyone have any suggestions? I can't try the fixes from this thread until I am able to check if they fix the problem or not. And I can't do this in prod.

[apsi15]

Same issue here. I am deploying both the function app and it's staging deployment slot through terraform using the Elastic Premium Linux. The deployment slot is working fine, but somehow the production slot is not able to reach the storage account. Disabling the vnet integration and network deny in the storage account bring the function back online. The settings are the same as @ketaanhshah posted. The NSG, Service Endpoints are configured correctly (tested it with SSH tcpping). I feel like the CONTENTSHARE is not suitable for the scenario where the SA configured with deny rule, or at least not suitable for File Share, since the WebJobsStorage configuration is reachting the blobs behind a firewall.