ObjectDisposedException at Orleans.Runtime.GrainTimer.ForwardToAsyncCallback

[alrz]

Caught and ignored exception thrown from timer callback for timer GrainTimer.sys.svc.user.95B6CAE6/10.42.89.234:11111@75196383Timer TimerCallbackHandler:[SystemTarget: S10.42.89.234:11111:75196383/sys.svc.user.95B6CAE6/10.42.89.234:11111@75196383@0d73e76f00000000f69b786b00000000]->System.Threading.Tasks.Task <Start>b__11_0(System.Object)

System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetServiceOrCreateInstance[T](IServiceProvider provider)
   at Orleans.Serialization.GeneratedCodeHelpers.OrleansGeneratedCodeHelper.GetService[TService](Object caller, ICodecProvider codecProvider) in /_/src/Orleans.Serialization/GeneratedCodeHelpers/OrleansGeneratedCodeHelper.cs:line 72
   at OrleansCodeGen.Orleans.Runtime.Proxy_IManagementGrain..ctor(GrainReferenceShared arg0, IdSpan arg1) in /_/src/Orleans.Core/Orleans.CodeGenerator/Orleans.CodeGenerator.OrleansSerializationSourceGenerator/Orleans.Core.orleans.g.cs:line 1115
   at Proxy_IManagementGrain(Object, GrainReferenceShared, IdSpan)
   at Orleans.GrainReferences.GrainReferenceActivatorProvider.GrainReferenceActivator.CreateReference(GrainId grainId) in /_/src/Orleans.Core/GrainReferences/GrainReferenceActivator.cs:line 380
   at Orleans.GrainReferences.GrainReferenceActivator.CreateReference(GrainId grainId, GrainInterfaceType interfaceType) in /_/src/Orleans.Core/GrainReferences/GrainReferenceActivator.cs:line 57
   at Orleans.GrainFactory.GetGrain(Type interfaceType, IdSpan grainKey, String grainClassNamePrefix) in /_/src/Orleans.Core/Core/GrainFactory.cs:line 217
   at Orleans.GrainFactory.GetGrain[TGrainInterface](Int64 primaryKey, String grainClassNamePrefix) in /_/src/Orleans.Core/Core/GrainFactory.cs:line 51
   at OrleansDashboard.SiloGrainService.CollectStatistics(Boolean canDeactivate)
   at Orleans.Runtime.GrainTimer.ForwardToAsyncCallback(Object state) in /_/src/Orleans.Runtime/Timers/GrainTimer.cs:line 116

[alrz]

With a slightly different stack from https://github.com/OrleansContrib/OrleansDashboard/issues/422

[EricSites]

I am getting the same thing on shutdown, endless loop of every second and the process will not close.

This is after doing a ctrl+c.

System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetServiceOrCreateInstance[T](IServiceProvider provider)
   at Orleans.Serialization.GeneratedCodeHelpers.OrleansGeneratedCodeHelper.GetService[TService](Object caller, ICodecProvider codecProvider) in /_/src/Orleans.Serialization/GeneratedCodeHelpers/OrleansGeneratedCodeHelper.cs:line 57
   at OrleansCodeGen.Orleans.Runtime.Proxy_IManagementGrain..ctor(GrainReferenceShared arg0, IdSpan arg1) in /_/src/Orleans.Core/Orleans.CodeGenerator/Orleans.CodeGenerator.OrleansSerializationSourceGenerator/Orleans.Core.orleans.g.cs:line 1115
   at Proxy_IManagementGrain(Object, GrainReferenceShared, IdSpan)
   at Orleans.GrainReferences.GrainReferenceActivator.CreateReference(GrainId grainId, GrainInterfaceType interfaceType) in /_/src/Orleans.Core/GrainReferences/GrainReferenceActivator.cs:line 57
   at Orleans.GrainFactory.GetGrain(Type interfaceType, IdSpan grainKey, String grainClassNamePrefix) in /_/src/Orleans.Core/Core/GrainFactory.cs:line 217
   at Orleans.GrainFactory.GetGrain[TGrainInterface](Int64 primaryKey, String grainClassNamePrefix) in /_/src/Orleans.Core/Core/GrainFactory.cs:line 51
   at OrleansDashboard.SiloGrainService.CollectStatistics(Boolean canDeactivate)
   at Orleans.Runtime.GrainTimer.ForwardToAsyncCallback(Object state) in /_/src/Orleans.Runtime/Timers/GrainTimer.cs:line 116

And

[01:49:15 INF] Orleans Silo stopped. <s:Orleans.Hosting.SiloHostedService>
[01:49:15 DBG] Hosting stopped <s:Microsoft.Extensions.Hosting.Internal.Host>
[01:49:15 ERR] Failed to address message Request [S127.0.0.1:11111:77262486 sys.client/hosted-127.0.0.1:11111@77262486]->[ pubsubrendezvous/SIGNALR_ORLEANS_STREAM_PROVIDER/SERVER_STREAM/64a76cf87e714ed5bb0dafe9a0c7751b] Orleans.Streams.IPubSubRendezvousGrain.GetAllSubscriptions(SIGNALR_ORLEANS_STREAM_PROVIDER/SERVER_STREAM/64a76cf87e714ed5bb0dafe9a0c7751b, sys.client/hosted-127.0.0.1:11111@77262486+8c415104af7943349707283505805c2e) #9041 <s:Orleans.Messaging>
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetRequiredKeyedService(Type serviceType, Object serviceKey)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
   at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 371
   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471
[01:49:15 ERR] Failed to address message Request [S127.0.0.1:11111:77262486 sys.client/hosted-127.0.0.1:11111@77262486]->[ serverdirectory/0] SignalR.Orleans.Core.IServerDirectoryGrain.Unregister(64a76cf8-7e71-4ed5-bb0d-afe9a0c7751b) #9040 <s:Orleans.Messaging>
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetRequiredKeyedService(Type serviceType, Object serviceKey)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
   at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 371
   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471

[ReubenBond]

I believe this has been fixed in main (there are nightly builds if you'd like to try those)

[ReubenBond]

Closing for now, since this ought to be fixed. Please open a new issue referencing this if you still see this after upgrading to Orleans v8.2.0