[Premium] Azure Function on a VNET can't start because of no access to zip using WEBSITE_RUN_FROM_PACKAGE

[pascalnaber]

Describe the bug When on a premium plan, and on a VNET, where the azure function is deployed using WEBSITE_RUN_FROM_PACKAGE. The Azure function cannot start. It has no access to the zip configured in WEBSITE_RUN_FROM_PACKAGE.

We have tried whitelisting the IP addresses of the azure function. But this does not work.

When we create a separate storage account which is not on a vnet. Only to make the zip available. Then configure WEBSITE_RUN_FROM_PACKAGE to get the zip from this location. Then the azure function is able to download the zip and runs fine.

To Reproduce Steps to reproduce the behavior: Run an azure function on a premium plan, use a vnet and deploy the azure function using WEBSITE_RUN_FROM_PACKAGE.

Expected behavior The azure function should be able to download the zip.

Additional context We are going to try to deploy using an alternative. Like with a Docker container. If there are other alternatives to deploy the azure function. please let us know.

[mivano]

Log lines shown in issue #1377

[sijucm]

I'm facing the same issue. It is now 1 year and there is no microsoft answers for this issue.

[v-bbalaiagar]

Apologies for the delayed response, the issue was somehow lost in the trace. We will check for the possibilities internally and update you with the findings.

[pragnagopa]

Tagging @sidkri @pgombar @balag0

[balag0]

@pascalnaber @mivano @sijucm This is on Linux or Windows?

[pragnagopa]

Please see https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-vnet#lock-down-your-storage-account and verify if your setup.

[yutao-huang]

Please see https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-vnet#lock-down-your-storage-account and verify if your setup.

@pragnagopa - could you verify this would actually solve the problem? It seems once a subnet is used for the private endpoint, it cannot be integrated to the function app anymore (showing "subnet must be empty and not already delegated" when trying to add a function app to it).

So my steps are:

• Added a private endpoint connection based off subnet default of my vent for the storage account blob resource that contains the function app package.
• Set the WEBSITE_RUN_FROM_PACKAGE setting of my function app to point to the package URL of the above blob with its private link.
• Tried adding VNet integration for the function app but noticed that the default subnet is disabled for selection, saying subnet must be empty and not already delegated.

My understanding is that in order for the function app to access the blob private link url, it has to be added to the same subnet of the private link, right? But right now, it's impossible to add it to a subnet that has been associated with a private link.

BTW, my subnet does contain NSG, which is required by our scenario.

[yutao-huang]

I also tried the below, but the function is still not working:

• Use firewall rules instead of NSG
• Add the blob URL host to the firewall FQDN rule allow list
• Add WEBSITE_DNS_SERVER=168.63.129.16 to the function app configuration

It seems the package is loaded because the Functions page can correctly list the function included in the package and I am able to see the function code in the function editor on the Code + Test page.

However, executing against the function URL is always getting 500 internal server error. The Logs pane below the editor shows the follow error (in HTML):

Server Error in '/' Application.
The format of the specified network name is invalid.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.IO.IOException: The format of the specified network name is invalid.


Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:


[IOException: The format of the specified network name is invalid.
]
   System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) +12601718
   System.IO.Directory.InternalCreateDirectory(String fullPath, String path, Object dirSecurityObj, Boolean checkHost) +12404913
   System.IO.Directory.InternalCreateDirectoryHelper(String path, Boolean checkHost) +88
   System.IO.Directory.CreateDirectory(String path) +34
   System.IO.Abstractions.DirectoryWrapper.CreateDirectory(String path) +10
   Kudu.Core.Infrastructure.FileSystemHelpers.CreateDirectory(String path) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\FileSystemHelpers.cs:32
   Kudu.Core.Infrastructure.FileSystemHelpers.EnsureDirectoryIgnoreAccessExceptions(String path) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\FileSystemHelpers.cs:48
   Kudu.Core.Environment.get_DeploymentsPath() in C:\Kudu Files\Private\src\master\Kudu.Core\Environment.cs:181
   Kudu.Services.Web.App_Start.NinjectServices.GetSettingsPath(IEnvironment environment) in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:795
   Kudu.Services.Web.App_Start.NinjectServices.EnsureValidDeploymentXmlSettings(IEnvironment environment) in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:0
   Kudu.Services.Web.App_Start.NinjectServices.RegisterServices(IKernel kernel) in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:154
   Kudu.Services.Web.App_Start.NinjectServices.CreateKernel() in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:131
   Ninject.Web.Common.Bootstrapper.Initialize(Func`1 createKernelCallback) +16
   Kudu.Services.Web.App_Start.NinjectServices.Start() in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:97

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
   System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
   System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +91
   System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +105
   WebActivatorEx.BaseActivationMethodAttribute.InvokeMethod() +73
   WebActivatorEx.ActivationManager.RunActivationMethods(Boolean designerMode) +637
   WebActivatorEx.ActivationManager.Run() +84

[InvalidOperationException: The pre-application start initialization method Run on type WebActivatorEx.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..]
   System.Web.Compilation.BuildManager.InvokePreStartInitMethodsCore(ICollection`1 methods, Func`1 setHostingEnvironmentCultures) +646
   System.Web.Compilation.BuildManager.InvokePreStartInitMethods(ICollection`1 methods) +147
   System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded) +107
   System.Web.Compilation.BuildManager.ExecutePreAppStart() +165
   System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +590

[HttpException (0x80004005): The pre-application start initialization method Run on type WebActivatorEx.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..]
   System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +10087352
   System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +99
   System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +263

Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.8.4330.0

I'm getting exactly the same error when accessing the Kudu site.

[pragnagopa]

@divyagandhisethi - please take a look. If docs need to be updated - please work with @cachai2

[MartinX3]

It is also confusing for me. The docs doesn't help me.

I want to connect two Web Apps via the same VNet. Web App A should be accessible from Internet and has access to the VNet. Web App B should only be accessible from the VNet.

So I added a private Endpoint for the inbound traffic to the VNet for Web App B.

But if I want to add a VNet integration for outbound traffic in Web App A it only tells me "subnet must be empty and not already delegated".

If I add a private endpoint to Web App A as well they can't use the 10.x addresses to communicate.

I don't understand how a private VNet communitation between two Web Apps should work.

[cutecycle]

Question for everyone: do you have "Outbound internet traffic" (vnetRouteAll) enabled on your virtual network integration?

[im-samz]

Hello all, I hope my responses can help:

@pascalnaber, @sijucm, if you want your function app to communicate to your storage account that has restricted traffic to a vnet, you'll need to ensure the following settings are set:

• vnetContentShareEnabled = true <-- this tells your function app to attempt connections to its storage account via the virtual network; otherwise, it will attempt to connect over the internet, which the storage account will block. In the Portal, you can find that by going to your Function App > Networking > Outbound traffic configuration > Virtual Network Integration > [your virtual network]. In the new page, check the box for "Content Storage".
• The subnet has a service endpoint to Microsoft.Storage. In Portal, you can do that by going to your Virtual Network > Subnets > [your specific subnet] > Service Endpoints. There should be a dropdown to select a service endpoint to Microsoft.Storage.

@yutao-huang, the subnet containing the private endpoint and the subnet that your function app is integrated with should be two different subnets. The subnet you integrate your Function App with must be empty because it needs to be delegated to Microsoft.Web. You'll be ok if both subnets are in the same VNet - just ensure that the DNS checks out. Azure Private DNS Zone should configure it for you if you select that option when creating the private endpoint in the Portal.

@MartinX3, similar to my response to @yutao-huang, can you try using two different subnets in the same VNet?

[amoghnatu]

We're also running into the same problem. Our OS is Linux and code package is a java package, uploaded to a storage account that is separate from the runtime storage account of functions. SKU is PremiumV3.

Is anyone looking into this issue?

[im-samz]

Hi @amoghnatu, the PremiumV3 SKU supports vnet integration. To have your function app fetch the zip package from the network secured storage account, you'll need to ensure that:

1. The function app is vnet integrated.
2. The storage account has allow-listed inbound traffic from that specific vnet + subnet.