Triggers transform to Error state - Cannot access a disposed object. Object name: 'IServiceProvider'

[DorSedoff]

Describe the bug

In some rare occasion we are getting the following exception: Quartz.SchedulerException: Problem instantiating type '<Job full name>': Cannot access a disposed object. Object name: 'IServiceProvider' This will transform the trigger into an Error state which will cause it cease firing... We are using the MicrosoftDependencyInjectionJobFactory and noticed it is happening when the service provider tried to create a scope (Even before it started to initialize the Job itself). So it tries to use a disposed IServiceProvider which shouldn't occur in the first place, I dont have clear idea why it is getting disposed, but I guess it could happen in some rare cases, like right before restarting a pod. In any case such scenarios shouldn't transfer a trigger to an Error state since the initialization of the job was not even started.

Version used

3.4

To Reproduce

not easy one since it is rare, the obvious option is to dispose the service provider while Quartz is running.

Expected behavior

Not transforming the trigger state to Error

Additional context Here is the stuck trace:

Quartz.SchedulerException: Problem instantiating type 'SchedulerService.Scheduler.QuartzNet.Jobs.AutoSyncImport.AutoSyncImportJob: Cannot access a disposed object.
Object name: 'IServiceProvider'.'
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.CreateScope(IServiceProvider provider)
   at Quartz.Simpl.MicrosoftDependencyInjectionJobFactory.InstantiateJob(TriggerFiredBundle bundle, IScheduler scheduler)
   at Quartz.Simpl.PropertySettingJobFactory.NewJob(TriggerFiredBundle bundle, IScheduler scheduler)
   at Quartz.Core.JobRunShell.Initialize(QuartzScheduler sched, CancellationToken cancellationToken)
   --- End of inner exception stack trace --- [See nested exception: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.CreateScope(IServiceProvider provider)
   at Quartz.Simpl.MicrosoftDependencyInjectionJobFactory.InstantiateJob(TriggerFiredBundle bundle, IScheduler scheduler)
   at Quartz.Simpl.PropertySettingJobFactory.NewJob(TriggerFiredBundle bundle, IScheduler scheduler)
   at Quartz.Core.JobRunShell.Initialize(QuartzScheduler sched, CancellationToken cancellationToken)]

Our setup:

            services.AddQuartz(q =>
            {
                q.UseSimpleTypeLoader();
                q.UseDefaultThreadPool(tp =>
                {
                    tp.MaxConcurrency = 10;
                });

                q.UseMicrosoftDependencyInjectionScopedJobFactory(options =>
                {
                    options.AllowDefaultConstructor = true;
                });
            });
            
            services.AddQuartzServer(options =>
            {
                options.WaitForJobsToComplete = true;
            });

[lahma]

Did you notice the issue template? Maybe you could follow the format and add information like the version you are using.

[DorSedoff]

Strange it was not opening the template before, Ill edit that, ty

[lahma]

The puzzling part is that service container is getting gracefully disposed, but Quartz is not getting notified about the shutdown.

[DorSedoff]

Indeed, its ether solving this puzzle or at least make this scenario transient to errors...

[lahma]

Would you like to offer a PR?

[DorSedoff]

Hey, not quit familiar with the source TBH, but I'll try to seat on that when Ill have some spare time :)

[lahma]

I think the key here is to check how call site (JobRunShell.Initialize) handles the call, I think it's expecting SchedulerException to happen in order it be a "graceful error". Originally it was pretty much no-go if you cannot call a constructor for an object type,

[lahma]

Looking at the code again, I think changing exception type won't help here. Would really require a proper detection that host is going down and should not even try to do anything.

[DorSedoff]

Hopefully I'll have some time handling this and reproducing the next week, thanks!

[A9G-Data-Droid]

I was seeing this issue when running some integration tests with Quartz. Each test would work on it's own but if I ran the entire test suite, I would get one success followed by one failure.

The solution was to add await testIScheduler.Shutdown(); at the bottom of each test. This says to me that there are some static things that you have to wait for cleanup before making a new scheduler. Otherwise you have a race condition between one shutdown and another instantiation, they can cross paths.