azure-functions-host icon indicating copy to clipboard operation
azure-functions-host copied to clipboard

Published 20 hours ago • verified publisher icon Azure

DryIoc.ContainerException causes lots of azure function to fail few times every day

Open abhiyadav1323 opened this issue 5 years ago • 14 comments

We see few DryIoc.ContainerException failures throughout the day in multiple azure functions in both our test and prod bed. So, I am hoping to get some guidance on figuring out why these failures are happening and what can we do to stop/reduce them.

Investigative information

  • Timestamp: 2020-09-08T22:13:12.0028507Z
  • Invocation ID: 5f7fe852-f2f8-45ae-b6bf-84478fd64b71
  • Region: West US 2

Repro steps

These functions are either timer triggered or queue triggered.

Actual behavior

We are seeing lots of azure functions failing in our test code few times a day with DryIoc.ContainerException.

Message:

Container is disposed and should not be used: Container is disposed. You may include Dispose stack-trace into the message via: container.With(rules => rules.WithCaptureContainerDisposeStackTrace())

Call Stack:

DryIoc.ContainerException:
   at DryIoc.Throw.It (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 8990)
   at DryIoc.Container.ThrowIfContainerDisposed (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 410)
   at DryIoc.Container.WithCurrentScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 436)
   at DryIoc.ResolverContext.OpenScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 2699)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.WebHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\WebHostServiceProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 51)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver.CreateChildScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\ScopedResolver.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 42)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 104)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInstanceWrapper.get_InstanceServices (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInstanceWrapper.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 44)
   at Microsoft.Azure.WebJobs.Host.Executors.DefaultJobActivator.CreateInstance (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\DefaultJobActivator.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 32)
   at Microsoft.Azure.WebJobs.Host.Executors.ActivatorInstanceFactory`1+<>c__DisplayClass1_1.<.ctor>b__0 (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\ActivatorInstanceFactory.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 20)
   at Microsoft.Azure.WebJobs.Host.Executors.ActivatorInstanceFactory`1.Create (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\ActivatorInstanceFactory.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 26)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.CreateInstance (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 44)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ParameterHelper.Initialize (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 846)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsyncCore>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 122)

Related information

  • Programming language used: C#, .net core

abhiyadav1323 avatar Sep 09 '20 00:09 abhiyadav1323

Hi @pragnagopa, @brettsam, @fabiocav , would it be possible for you guys to provide a brief explanation as to why this is happening? We have an azure function based infrastructure for Microsoft Sports and we see this exception almost every day across all azure functions across all apps in both test and prod environment which sometimes causes us to miss updates during live games.

abhiyadav1323 avatar Sep 12 '20 00:09 abhiyadav1323

@abhiyadav1323 this could happen for a few different reasons, so it's hard to diagnose without looking more closely at the details of how you're using the services. The vast majority of cases with this issue we review are caused by function code holding onto services beyond their lifetime scope (e.g. a function execution caching a service/factory or the service provider across different invocations).

fabiocav avatar Sep 14 '20 17:09 fabiocav

@fabiocav We do use DI singletons for cosmos db clients, http client factory, etc. as per recommendations in the Azure Functions documentation. What information do you need to debug this further?

zmarty avatar Sep 14 '20 19:09 zmarty

This also looks to be related : https://github.com/Azure/azure-functions-host/issues/5240

uprightbass360 avatar Sep 18 '20 19:09 uprightbass360

We had the same issue and on top of that function timer trigger was not working properly either. Initially this exception was triggering during resolution of IHttpClient and so I ended up looking into that which was in the wrong direction.

The host is disposed and cannot be used. Disposed object: 'Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver' Container is disposed and should not be used: Container is disposed.
You may include Dispose stack-trace into the message via:
container.With(rules => rules.WithCaptureContainerDisposeStackTrace()) 

Microsoft.Azure.WebJobs.Script.HostDisposedException:
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:110)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.CreateHandlerEntry (Microsoft.Extensions.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory+<>c__DisplayClass14_0.<.ctor>b__1 (Microsoft.Extensions.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Lazy`1.ViaFactory (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Lazy`1.CreateValue (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Lazy`1.CreateValue (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Lazy`1.get_Value (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.CreateHandler (Microsoft.Extensions.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.CreateClient (Microsoft.Extensions.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Inner exception DryIoc.ContainerException handled at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope:
   at DryIoc.Throw.It (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:8993)
   at DryIoc.Container.ThrowIfContainerDisposed (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:411)
   at DryIoc.Container.WithCurrentScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:437)
   at DryIoc.ResolverContext.OpenScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:2702)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.WebHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\WebHostServiceProvider.cs:51)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver.CreateChildScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\ScopedResolver.cs:43)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:106)

But finally, adding the configuration setting AzureFunctionsWebHost__hostId solved 90% of these errors. You can read more here.

But for the remaining 10% of the cases, I had to debug the framework code because it wasn't telling me what object it was trying to resolve. Luckily during call-stack unwinding process of the exception, I was able to see the local variables and saw the object. I had to then change the DI code in StartUp/Program class. There were some singleton objects that we new-ed up and resolved using syntax like x => x.GetService but I replaced it with new-ed up object instead.

Hopefully this would help others.

vsha041 avatar Nov 26 '21 17:11 vsha041

@vsha041 We are facing similar exceptions(DryIoc.ContainerException) for our Azure function app (v4) even after we added the AzureFunctionsWebHost__hostId, Can you share a little bit more about the DI issue and how you fixed it.

Shuvajyoti avatar May 12 '22 15:05 Shuvajyoti

@Shuvajyoti - it was something along these lines but I will re-check again and let you know next week. By the way make sure the azure function configuration variable AzureFunctionsWebHost__hostId is set to lowercase GUID without the dashes. E.G. = 34c63f7c44ee4271ba7ffa02c8c883a9

Before

services.AddScoped(x => new P());
services.AddScoped(x => 
{
  x.propA = x.getService<P> (); // this line was failing to resolve after upgrading to V4.
});

After

var p = new P();
services.AddScoped(x => p);
services.AddScoped(x => 
{
  x.propA = p;
});

vsha041 avatar May 13 '22 10:05 vsha041

@vsha041 I see the same exception how do I update my code below:

public override void Configure(IFunctionsHostBuilder builder)
{
            base.Configure(builder);
            builder.Services.AddSingleton<IServiceClient>((services) =>
            {
                return new ServiceClient(CreateAuthProviderFromSecret(value));
            })
            .AddScoped<IGraphRepository, GraphRepository>()            
            .AddSingleton<IBlobRepository, BlobRepository>((s) =>
            {
                var configuration = s.GetService<IConfiguration>();
                var a = configuration["a"];                
                return new BlobRepository(a);
            });
}

lipalath-ms avatar Aug 12 '22 21:08 lipalath-ms

@ilantom you can try this:

public override void Configure(IFunctionsHostBuilder builder)
{
        base.Configure(builder);
        var configuration = builder.GetContext().Configuration;
        var a = configuration["a"];
        var b = new BlobRepository(a);

        builder.Services
            .AddSingleton<IServiceClient>(_ => new ServiceClient(CreateAuthProviderFromSecret(value)))
            .AddScoped<IGraphRepository, GraphRepository>()
            .AddSingleton<IBlobRepository, BlobRepository>(_ => b);
}

vsha041 avatar Aug 13 '22 02:08 vsha041

@ilantom - also make sure you add the configuration variable AzureFunctionsWebHost__hostId in your function and give it a value of lower case GUID and without the dashes (e.g. 1835d7b55c984790815d072cc94c6f71) Untitled

vsha041 avatar Aug 13 '22 02:08 vsha041

@vsha041 Thank you! I use deployment slots in Azure Function. Should I add different GUID values of AzureFunctionsWebHost__hostId for both staging and production slots?

lipalath-ms avatar Aug 15 '22 19:08 lipalath-ms

@ilantom - yes but GUID should be different for both slots unless staging one is deleted immediately after production deployment. So if you have multiple active slots at any given time then they all should have different GUID.

vsha041 avatar Aug 15 '22 20:08 vsha041

Ok let me try that. Thank you! Also, while swapping slots, should these values be swapped?

lipalath-ms avatar Aug 15 '22 20:08 lipalath-ms

@vsha041 Just to be clear this is how my startup file looks like:

        public override void Configure(IFunctionsHostBuilder builder)
        {
            base.Configure(builder);

            builder.Services.AddSingleton<IGraphServiceClient>((services) =>
            {
                return new GraphServiceClient(FunctionAppDI.CreateAuthProviderFromSecret(services.GetService<IOptions<GraphCredentials>>().Value));
            })
            .AddScoped<IGraphGroupRepository, GraphGroupRepository>()
            .AddScoped<Calculator>()
            .AddSingleton<IBlobStorageRepository, BlobStorageRepository>((s) =>
            {
                var configuration = s.GetService<IConfiguration>();
                var storageAccountName = configuration["storageAccountName"];
                var containerName = configuration["containerName"];

                return new BlobStorageRepository($"https://{storageAccountName}.blob.core.windows.net/{containerName}");
            });
        }

Could you please let me know how to update this?

lipalath-ms avatar Aug 15 '22 21:08 lipalath-ms

Ok let me try that. Thank you! Also, while swapping slots, should these values be swapped?

It's ok if they are swapped as long as they are different. Or you can put slotSettings: true if you don't want them to get swapped.

vsha041 avatar Aug 15 '22 22:08 vsha041

@vsha041 Just to be clear this is how my startup file looks like:

        public override void Configure(IFunctionsHostBuilder builder)
        {
            base.Configure(builder);

            builder.Services.AddSingleton<IGraphServiceClient>((services) =>
            {
                return new GraphServiceClient(FunctionAppDI.CreateAuthProviderFromSecret(services.GetService<IOptions<GraphCredentials>>().Value));
            })
            .AddScoped<IGraphGroupRepository, GraphGroupRepository>()
            .AddScoped<Calculator>()
            .AddSingleton<IBlobStorageRepository, BlobStorageRepository>((s) =>
            {
                var configuration = s.GetService<IConfiguration>();
                var storageAccountName = configuration["storageAccountName"];
                var containerName = configuration["containerName"];

                return new BlobStorageRepository($"https://{storageAccountName}.blob.core.windows.net/{containerName}");
            });
        }

Could you please let me know how to update this?

You can try below code.

public override void Configure(IFunctionsHostBuilder builder)
        {
            base.Configure(builder);
	// 1) get graph credentials here i.e. read from appSettings here
            builder.Services.AddSingleton<IGraphServiceClient>((services) =>
            {
                return new GraphServiceClient(FunctionAppDI.CreateAuthProviderFromSecret(graphCredentials));
            })
            .AddScoped<IGraphGroupRepository, GraphGroupRepository>()
            .AddScoped<Calculator>()
			
	// 2) get configuration object here 
	// var configuration = builder.GetContext().Configuration;
            .AddSingleton<IBlobStorageRepository, BlobStorageRepository>((s) =>
            {                
                var storageAccountName = configuration["storageAccountName"];
                var containerName = configuration["containerName"];

                return new BlobStorageRepository($"https://{storageAccountName}.blob.core.windows.net/{containerName}");
            });
        }

vsha041 avatar Aug 15 '22 22:08 vsha041

Update 2022

We faced this problem and found out that AzureFunctionsWebHost__hostid was missing in this particular function and we have added it manually using a new guid and the function was working as expected

gethari avatar Dec 27 '22 07:12 gethari

Update 2022

We faced this problem and found out that AzureFunctionsWebHost__hostid was missing in this particular function and we have added it manually using a new guid and the function was working as expected

Thanks for verifying and confirming it.

vsha041 avatar Dec 27 '22 17:12 vsha041