[Premium] Can't deploy Function App to storage account with VNET restriction

[ishepherd]

Below is an ARM template which creates:

1. Microsoft.Storage/storageAccounts
   • A storage account with a VNET ACL (virtualNetworkRules)
2. Microsoft.Web/Sites
   • A function app using that storage account for AzureWebJobsStorage and WEBSITE_CONTENTAZUREFILECONNECTIONSTRING
3. Microsoft.Web/Sites/config
   • VNET integration for the function app

Resource 2 fails with

There was a conflict. The remote server returned an error: (403) Forbidden.

TemplateAndError.zip

I believe the Microsoft.Web ARM provider is attempting connection to the storage account and can't. (I enabled "Allow trusted Microsoft services to access this storage account" but that isn't helping.)

An obvious workaround is to take the ACL out of the storage account, and deploy it a second time later with the ACL applied.

Is there a cleaner workaround? It's a common practice for people to deploy the Function App and Storage Account all in the same template.

See also: https://github.com/Azure/azure-functions-ux/issues/2279

[marcin-vt]

Does restricting access to storage account after deployment of ARM template work? Or it will generate more errors during function deployment (az functionapp deployment....)?

[jeffhollan]

A bit of a longer answer to explain what the current state is today, and options for deploying code to an app with VNet restrictions. This is all assuming the Azure Functions Premium Plan which supports the regional VNet.

The Premium Plan has VNet support which enables the function app itself to make secure connections to resources restricted to a private network. This includes making a call to a resource within a VNet (like a VM), or a service that has service endpoints enabled (like storage). Calls that are made during the execution of an app should work just fine with the out of the box VNet support.

There are some layers outside of the function app itself (so outside of your "container" running the app, but on the underlying service infrastructure) that may also need access to a resource behind a VNet. One of the most common examples is for a trigger. If you wanted to trigger on a storage account for example, we have pieces of the service (called the event driven scale controller) that will monitor the storage account and scale based on the number of events. By default, that component runs outside of your app, but also needs access to your VNet. However, we have since added a feature that lets you toggle so that those checks happen within the context of your app. It's documented here. More or less how it works is our infrastructure just asks your app "hey can you check this storage account for me for events, it has service endpoints so I can't see it" and your app makes the secure call over VNet and returns the info. You can opt into that settings.

That's a long intro of saying there are still a few other pieces of the service outside of your "app" Vm/container. One of them is the file system that hosts your code and then mounts that code to your "container." Today, we don't have a way for that file system to mount a source that is behind a VNet. The file system that has your code specifically is the WEBSITE_CONTENTAZUREFILECONNECTIONSTRING setting that @ishepherd. We are working with the team to understand how we can enable that call to happen securely through the user VNet. In the meantime, you can deploy a Premium Plan without this setting (not defining a WEBSITE_CONTENTAZUREFILECONNECTIONSTRING or WEBSITE_CONTENTSHARE setting). This will mean we use internal storage to store the code - so you don't have to worry about VNets. ⚠️ however, doing this will limit your premium plan to only scale out to 20 instances maximum ⚠️ . But if that's ok for your app, it's probably the best option we have for you now.

All of the CONTENT settings are set at time of app create, so @ishepherd I'm not sure but I suspect the reason it appears to work if you apply ACLs later may be just a false flag and I'm not sure if it will work after you deploy more and more code or it scales out beyond 20. I'm not positive though, I personally still have a few questions around how much things like https://docs.microsoft.com/en-us/azure/azure-functions/run-functions-from-deployment-package#adding-the-website_run_from_package-setting could be used in the interim. Met with the team on this scenario before the holiday break and will circle back now for any updates.

[marcin-vt]

Thanks for taking this up @jeffhollan.

However removing WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_CONTENTSHARE from my function configuration caused some errors after restart.

In the azure portal I see now:

Azure Functions Runtime is unreachable. Click here for details on storage configuration.

When trying to deploy function from commandline (az functionapp deployment source config-zip):

Getting scm site credentials for zip deployment Starting zip deployment. This operation can take a while to complete ... Deployment endpoint responses with status code 500 Deployment status endpoint https://xxxx.scm.azurewebsites.net/api/deployments/latest returns malformed data. Retrying...

Also, since we use this function as a scheduler (to start container every hour), another variable (AzureWebJobsStorage) with storage account credentials has to be set.

[markusfoss]

The solution worked great for us @jeffhollan - we enhanced our ARM provisioning Azure DevOps release pipelines to not include WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_CONTENTSHARE . Then our vnet connected premium function is provisioned as expected.

HOWEVER: Docs in https://docs.microsoft.com/en-us/azure/azure-functions/functions-app-settings should be updated to reflect this issue (it also applies when you have a open VNET connected to your function app). It should clearly state the impact - it cost us a day of troubleshooting (it worked when I deployed from Visual Studio 2019, but not when we used our release pipelines)

[dearsi-mocha]

The only problem I see when removing the settings WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_CONTENTSHARE is that the host keys are removed, even attempting to create keys will give you a bad request

[erwelch]

@jeffhollan - We're currently using WEBSITE_RUN_FROM_PACKAGE with a ZipDeploy from ARM that provides a URL to blob storage where the package is. When deploying after removing WEBSITE_CONTENTAZUREFILECONNECTIONSTRING, I'm getting 429 errors with the full error stack below. I read this can be from not having the WEBSITE_CONTENTAZUREFILECONNECTIONSTRING in this issue. I've made the blob storage public just to rule out that being the issue I'm running into. It's a premium function app. Regarding your questions about the WEBSITE_RUN_FROM_PACKAGE - was that meaning it won't work and I need to ZipDeploy without the WEBSITE_RUN_FROM_PACKAGE run from package set? Even without that I'm still seeing this error. Wondering if I need to deploy differently?

[IOException: The user name or password is incorrect.
]
   System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) +1091
   System.IO.Directory.InternalCreateDirectory(String fullPath, String path, Object dirSecurityObj, Boolean checkHost) +1394
   System.IO.Directory.InternalCreateDirectoryHelper(String path, Boolean checkHost) +92
   System.IO.Abstractions.DirectoryWrapper.CreateDirectory(String path) +14
   Kudu.Core.Infrastructure.FileSystemHelpers.EnsureDirectoryIgnoreAccessExceptions(String path) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\FileSystemHelpers.cs:48
   Kudu.Services.Web.App_Start.NinjectServices.GetSettingsPath(IEnvironment environment) in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:775
   Kudu.Services.Web.App_Start.NinjectServices.EnsureValidDeploymentXmlSettings(IEnvironment environment) in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:0
   Kudu.Services.Web.App_Start.NinjectServices.RegisterServices(IKernel kernel) in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:155
   Kudu.Services.Web.App_Start.NinjectServices.CreateKernel() in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:130
   Ninject.Web.Common.Bootstrapper.Initialize(Func`1 createKernelCallback) +23
   Kudu.Services.Web.App_Start.NinjectServices.Start() in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:95

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
   System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
   System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +260
   System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +142
   WebActivatorEx.BaseActivationMethodAttribute.InvokeMethod() +81
   WebActivatorEx.ActivationManager.RunActivationMethods(Boolean designerMode) +699
   WebActivatorEx.ActivationManager.Run() +120

[InvalidOperationException: The pre-application start initialization method Run on type WebActivatorEx.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..]
   System.Web.Compilation.BuildManager.InvokePreStartInitMethodsCore(ICollection`1 methods, Func`1 setHostingEnvironmentCultures) +903
   System.Web.Compilation.BuildManager.InvokePreStartInitMethods(ICollection`1 methods) +169
   System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded) +171
   System.Web.Compilation.BuildManager.ExecutePreAppStart() +172
   System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +848

[HttpException (0x80004005): The pre-application start initialization method Run on type WebActivatorEx.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..]
   System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +532
   System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +111
   System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +714

[dylan-asos]

Had a similar issue recently during provisioning a logic app v2 with services restricted by VNET + Service Endpoints. The answer from @jeffhollan was really helpful to understand some of what was going on and certainly put me in the right direction.

I'm provisioning all the infrastructure using Terraform and during that stage I'm restricting the newly created storage account to a VNET, creating a logic app and also associating it with the VNET. The logic app needs to configure itself with storage for WEBSITE_CONTENTAZUREFILECONNECTIONSTRING, WEBSITE_CONTENTSHARE and AzureWebJobsStorage - after provisioning all my dependencies, both the logic application and the kudu interface are unresponsive, returning 502\503s. . Behaviour seemed quite erratic here, I also experienced

Access to the path 'C:\home\site\wwwroot' is denied.
System.Private.CoreLib: Access to the path 'C:\home\site\wwwroot\host.json' is denied.

As per https://github.com/Azure/Azure-Functions/issues/1349#issuecomment-787492922

I had WEBSITE_VNET_ROUTE_ALL = 1 as I needed to route all outbound traffic via a NAT gateway and I set WEBSITE_CONTENTOVERVNET = 1, to no avail. The content from Jeff's point and the 'Access to path' error message pointed me to the storage account file share, which had not been created. Therefore, rather than leaving this for the runtime to create the share, I added an explicit file share creation to our provisioning script, i.e.

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_share#example-usage

Then, pass the file share name to the logic app for provisioning, so sequence of steps becomes

1. Create storage account
2. Generate file share unique name
3. Create file share in storage account
4. When provisioning logic app, pass name from 2) as parameter value for WEBSITE_CONTENTSHARE

[hecflores]

@dylan-asos I also am facing this issue but for function apps.

From what I read (More specifically from @jeffhollan comment) was that...

The root of the issue is that VNet Integration does not support Mounting a Drive, which is listed here.

Please anyone confirm the statement above...

If this is true, then how does what @dylan-asos did work? Or is it different for logic apps...

I currently am trying the approach that was shown to work for logic apps and applying it to function apps and seeing if that works.

Automation does the following...

1. Create Storage Account StorageV2
2. Create Private Endpoints (blob, file, dfs, web, table) for storage account
3. Create Share (unique name) - Share Name
4. Create Function App
   • WEBSITE_CONTENTAZUREFILECONNECTIONSTRING = Connection String of Storage Account
   • WEBSITE_CONTENTSHARE = Share Name
   • WEBSITE_VNET_ROUTE_ALL = 1
   • WEBSITE_DNS_SERVER = 168.63.129.16
   • WEBSITE_CONTENTOVERVNET = 1

After doing this, it was successful.

FYI, for durability function app I had to create the queue private endpoint which it does list in documentation.

[dylan-asos]

@hecflores how did you get on?

As per https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-vnet, one of the steps there is to create a file share, e.g. https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-vnet#create-a-file-share. The sequence of events listed there is essentially what I'm doing now.

[hecflores]

@dylan-asos I updated my comment. It was successful for me! :)

[sadiqkhoja]

@erwelch did you find any work around [IOException: The user name or password is incorrect.]

[johanclassoncab]

The suggestion to not define WEBSITE_CONTENTAZUREFILECONNECTIONSTRING or WEBSITE_CONTENTSHARE setting did not work for me. Zip deploying worked, but the functions never could start up.

What finally worked was setting WEBSITE_RUN_FROM_PACKAGE to an URL to a blob with SAS token. This also worked with blobs stored in a storage account only accepting traffic from VNET, where WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_RUN_FROM_PACKAGE=1 did not.

Note that docs warn that startup performance will be worse using URL instead of 1 (ref.).

[danielrobinson95]

@jeffhollan any update on when this issue will be resolved?

[wilcoblom]

@jeffhollan bump

[haciz]

Problem is still there even with manual creation of the resources, is there any update from MS when this going to be resolved, it seems as pretty old issue.

[rubenaster]

So, when using Terraform to create the File Share, the subnet for the DevOps build agent (that is running as a Linux VM with access to the VNet) needs to be included in the Storage allowed subnets, right? Otherwise I get an authorization error when the check for the existence of the File Share is being made:

Error: checking for existence of existing Storage Share "logicXXXXX-q01-content" (Account "stXXXXXq01" / Resource Group "RG-XXXXXX-Q01"): shares.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailure" Message="This request is not authorized to perform this operation.\nRequestId:94XXXXX00\nTime:2022-03-17T17:44:28.6416893Z"

If that's the case then this is pretty unsatisfying since allowing the DevOps agents subnet would only be needed for creating this File Share.