Upgrading to Microsoft.Azure.Functions.Extensions v1.1 breaks function in Azure

[mcalvin-stratus10]

Please provide a succinct description of the issue.

Repro steps

We upgraded to Microsoft.Azure.Functions.Extensions version 1.1 yesterday, and all of our Azure functions have the following error message:

System.Private.CoreLib: could not load type 'Microsoft.Azure.WebJobs.Hosting.IWebJobsStartup2' from Assembly Microsoft.Azure.WEbJobs.Host, Version=3.0.17.0 ....'

Expected behavior

The Azure Function to run in Azure like it does locally

Actual behavior

Get error message above.

Known workarounds

We are rolling back to v1.0 on all of our functions.

Related information

Provide any related information

• We are running Azure Functions as Linux Containers on EP1 tier

[LayZeeDK]

Probably related to https://github.com/Azure/azure-functions-dotnet-extensions/issues/51.

[aaltotsky]

I had the same issue: System.Private.CoreLib: Could not load type 'Microsoft.Azure.WebJobs.Hosting.IWebJobsStartup2' from assembly 'Microsoft.Azure.WebJobs.Host, Version=3.0.16.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

Add NuGet package <PackageReference Include="Microsoft.Azure.WebJobs" Version="3.0.19" /> rebuild solution Check the version of the Microsoft.Azure.WebJobs.Hostin C:\Users[User]\AppData\Local\AzureFunctionsTools\Releases\3.13.0\cli

I fixed it im my open source: https://github.com/RenetConsulting/angularcore.net/tree/master/Services/CoreCaptchaAzure

[brettsam]

@mcalvin-stratus10 -- maybe our container images weren't up-to-date when you tried moved forward. Have you been able to try again?

[DOliana]

We are having the same issue. Adding Microsoft.Azure.Webjobs 3.0.19 did not fix it for us. These are our packages:

        <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.9" />
        <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
        <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid" Version="2.1.0" />
        <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.1.1" />
        <PackageReference Include="NLog.Extensions.Logging" Version="1.6.5" />

What did work is to revert Microsoft.Azure.Functions.Extensions to 1.0.0

[brettsam]

@DOliana -- is that locally or in Azure? If locally, can you make sure you're running on the latest version of the Azure Functions Core Tools?

[DOliana]

It's locally when trying to debug in Visual Studio - so I am using what came with Visual Studio 16.7.5 (that's 3.0.2534). While trying to update I foudn out that I was facing the issue described here on SO.

After performing the described steps and veryfing that the used version of the core tools is newer (not the latest, but higher 3.0.2931) it worked again.

So @brettsam thank you for pointing me in the right direction.

[viniciuschiele]

I'm also facing this issue on Mac, I added the latest version of Microsoft.Azure.Webjobs and installed the latest version of Azure Functions Core Tools, no success. It does work when I run func start on the terminal.

[LayZeeDK]

I'm also facing this issue on Mac, I added the latest version of Microsoft.Azure.Webjobs and installed the latest version of Azure Functions Core Tools, no success. It does work when I run func start on the terminal.

Try updating all the dependencies and tools listed here: https://github.com/Azure/azure-functions-dotnet-extensions/issues/51#issuecomment-698359427

[mwwhited]

When I updated to Microsoft.Azure.Functions.Extensions 1.1.0 and deploy to azure I get an error about ...

Microsoft.Azure.WebJobs.Script: The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class.

It then lists every function that has a connection string, time or queue name that is configured versus being a constant in code.

[aaltotsky]

Check the version of the Microsoft.Azure.WebJobs.Host.dll in the C:\Users\CURRENT_USER\AppData\Local\AzureFunctionsTools\Releases\3.13.0\cli folder. My is 3.0.19.0

[Ruud-cb]

I'm having the same error as @mwwhited , I'm now trying to revert to v1.0.0. Above reference shows detailed azure function setup & versions whoever is interested. EDIT: Downgrade did solve the problem, still having issues regarding trigger syncing with azure infrastructure, which I am now writing a issue for.

[realrubberduckdev]

I just installed the latest azure-functions-core-tools and I do not get this load error anymore.

npm install -g azure-functions-core-tools@3

func --version
3.0.2996

[Ruud-cb]

@realrubberduckdev I didn't mention it that clearly, but my problem was occuring in Azure, did they update it there as well?

[IGx89]

Just in the past week we've start running into this same issue on our Isolated app service plan in Azure, just with a slightly older host version (3.0.16.0). It only happens when our pool auto-scales up, and only happens on one of the newly started instances, and resolves immediately once I restart the function app. We're going to start taking advantage of the new Health Check functionality to auto-restart our app when this happens, but obviously that's just a band-aid and not a real solution.

Why new instances in Azure are suddenly showing up running old function runtime versions makes no sense to me, and unfortunately there's little more I can do to troubleshoot a runtime issue like that.

[brettsam]

@IGx89 -- what version of the Functions Host is running in Azure for you? The Extensions 1.1.0 requires 3.0.18 or higher: https://www.nuget.org/packages/Microsoft.Azure.Functions.Extensions/1.1.0. Could you be building locally against a newer version, but deploying to an older?

[IGx89]

@brettsam How do I tell what version of the Functions Host my app is running on? It's not in my zip deploy archive, so it's part of the Azure environment itself. I have FUNCTIONS_EXTENSION_VERSION set to ~3, so I'm at the mercy of whatever Azure decides to use.

[brettsam]

If you could share your Function App name (either explicitly or privately), I can see if I find anything from the backend logs.

[IGx89]

2021-02-16T15:57:55.9558916Z baa8f447-4f6e-4168-a18e-61b45acfba9, East US. Last confirmed time this issue existed for us was ~1:00pm CST yesterday (when scaled up to four instances), though it's likely happened since too.

If there's a way for us to see the host version of a particular instance, using Kudu or whatever, I'd be very interested in that too. Then next time this happens I could look up the version of the various instances and confirm if this is the issue or not.

[IGx89]

@brettsam This just happened ~30min ago during an autoscale up -- found the details on the "Function Executions and Errors" report under "Availability and Performance":

Description: A host error has occurred during startup operation 'bfde5624-64ec-4c74-b482-08e116c17411'.
Last offline at:	2/16/2021 5:03:27 PM
Details:
System.TypeLoadException : Could not load type 'Microsoft.Azure.WebJobs.Hosting.IWebJobsStartup2' from assembly 'Microsoft.Azure.WebJobs.Host, Version=3.0.16.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.
  at System.Reflection.RuntimeAssembly.GetType(QCallAssembly assembly,String name,Boolean throwOnError,Boolean ignoreCase,ObjectHandleOnStack type,ObjectHandleOnStack keepAlive,ObjectHandleOnStack assemblyLoadContext)
  at System.Reflection.RuntimeAssembly.GetType(String name,Boolean throwOnError,Boolean ignoreCase)
  at Microsoft.Azure.WebJobs.Script.DependencyInjection.ScriptStartupTypeLocator.<>c__DisplayClass9_2.b__1(Assembly assembly,String typeName,Boolean ignoreCase) at D:\a\1\s\src\WebJobs.Script\DependencyInjection\ScriptStartupTypeLocator.cs : 144
  at System.TypeNameParser.ResolveType(Assembly assembly,String[] names,Func`4 typeResolver,Boolean throwOnError,Boolean ignoreCase,StackCrawlMark& stackMark)
  at System.TypeNameParser.ConstructType(Func`2 assemblyResolver,Func`4 typeResolver,Boolean throwOnError,Boolean ignoreCase,StackCrawlMark& stackMark)
  at System.TypeNameParser.GetType(String typeName,Func`2 assemblyResolver,Func`4 typeResolver,Boolean throwOnError,Boolean ignoreCase,StackCrawlMark& stackMark)
  at System.Type.GetType(String typeName,Func`2 assemblyResolver,Func`4 typeResolver,Boolean throwOnError,Boolean ignoreCase)
  at async Microsoft.Azure.WebJobs.Script.DependencyInjection.ScriptStartupTypeLocator.GetExtensionsStartupTypesAsync() at D:\a\1\s\src\WebJobs.Script\DependencyInjection\ScriptStartupTypeLocator.cs : 113
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at Microsoft.Azure.WebJobs.Script.DependencyInjection.ScriptStartupTypeLocator.GetStartupTypes() at D:\a\1\s\src\WebJobs.Script\DependencyInjection\ScriptStartupTypeLocator.cs : 55
  at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.UseExternalStartup(IWebJobsBuilder builder,IWebJobsStartupTypeLocator startupTypeLocator,ILoggerFactory loggerFactory) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 167
  at Microsoft.Azure.WebJobs.Script.ScriptHostBuilderExtensions.UseScriptExternalStartup(IWebJobsBuilder builder,ScriptApplicationHostOptions applicationHostOptions,ILoggerFactory loggerFactory,IExtensionBundleManager extensionBundleManager,IMetricsLogger metricsLogger) at D:\a\1\s\src\WebJobs.Script\ScriptHostBuilderExtensions.cs : 241
  at Microsoft.Azure.WebJobs.Script.ScriptHostBuilderExtensions.<>c__DisplayClass3_0.b__0(HostBuilderContext context,IWebJobsBuilder webJobsBuilder) at D:\a\1\s\src\WebJobs.Script\ScriptHostBuilderExtensions.cs : 119
  at Microsoft.Extensions.Hosting.WebJobsHostBuilderExtensions.<>c__DisplayClass4_0.b__1(HostBuilderContext context,IServiceCollection services) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsHostBuilderExtensions.cs : 47
  at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
  at Microsoft.Extensions.Hosting.HostBuilder.Build()
  at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptHostBuilder.BuildHost(Boolean skipHostStartup,Boolean skipHostConfigurationParsing) at D:\a\1\s\src\WebJobs.Script.WebHost\DefaultScriptHostBuilder.cs : 59
  at async Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation,Int32 attemptCount,JobHostStartupMode startupMode) at D:\a\1\s\src\WebJobs.Script.WebHost\WebJobsScriptHostService.cs : 234

[brettsam]

I'm seeing your app occasionally start up with a really old version of the host (from last April). I've not seen this before and I wonder if it has something to do with your ASE. I'm going to ask some experts on our team -- but your best bet may be to submit an support ticket (which should get routed to the correct team).

I'll update here if I hear back any recommendations...

[brettsam]

If you file a ticket, can you let me know the ticket number here? Then I can refer to it internally.

[IGx89]

Sure thing: 2102160040004856

Thanks a lot for your help with this!

[CarlosOnline]

We are still having this issue. Restarting helps, but this is not an acceptable solution. The repro happens with QueueTrigger functions with connection strings coming from key vault. The Azure Function is behind a VNet / ISE.

[qcnguyen]

I still encounter the issue Could not load type 'Microsoft.Azure.WebJobs.Hosting.IWebJobsStartup2' from assembly 'Microsoft.Azure.WebJobs.Host, Version=3.0.16.0 Our function is V3

This happen occasionally, Any solution for this problem please