Unable to find an IGrainContextActivatorProvider for grain type sys.client

[jsteinich]

I observed around 200k of these exceptions in about a 5 minute period earlier today. It then happened again about 30 minutes later (30k in about 2 minutes).

• The log message template is Error creating activation for grain {TargetGrain} (interface: {InterfaceType}). Message {Message}.

• The InterfaceType is Orleans.Streams.IStreamConsumerExtension.

• Example TargetGrain: sys.client/b6bc750f23ee452ead72a06eb3baddab+2bce487c693f45e2bbdf0e1a9f1af8dd (each burst seems to be the same grain)

• Example Message: Request [S10.52.8.158:22222:110528910 sys.svc.stream.agent/10.52.8.158:22222@110528910+MemProvider_5_memprovider-4-0x80000004]->[S10.52.11.131:22222:110550131 sys.client/b6bc750f23ee452ead72a06eb3baddab+2bce487c693f45e2bbdf0e1a9f1af8dd] Orleans.Streams.IStreamConsumerExtension.ErrorInStream(Orleans.Runtime.GuidId, System.Exception) #4712A6E508A75DA7

• Full exception: System.InvalidOperationException: Unable to find an IGrainContextActivatorProvider for grain type sys.client at Orleans.Runtime.GrainContextActivator.CreateActivator(GrainType grainType) in /_/src/Orleans.Runtime/Activation/IGrainContextActivator.cs:line 88 at Orleans.Runtime.GrainContextActivator.CreateInstance(GrainAddress address) in /_/src/Orleans.Runtime/Activation/IGrainContextActivator.cs:line 56 at Orleans.Runtime.Catalog.GetOrCreateActivation(GrainId& grainId, Dictionary`2 requestContextData, MigrationContext rehydrationContext) in /_/src/Orleans.Runtime/Catalog/Catalog.cs:line 170 at Orleans.Runtime.Messaging.MessageCenter.ReceiveMessage(Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 534

• We recently updated to Orleans 9.1.2 (from 8.2) and hadn't seen this error before.

• We use the memory provider for a few streams. The one most likely to have gotten messages has two types of consumers: 1. singleton on each silo; 2. singleton on each Orleans client

There were a few other logs right before the stream of errors that may be helpful:

1. Dropping client {ClientId}, {IdleDuration} after disconnect with no reconnect - ClientId = sys.client/b6bc750f23ee452ead72a06eb3baddab
2. 4x Dropping expired message {Message} at phase {Phase} - Phase = Send; Example Message = Transient Rejection (info: Client dropped Orleans.Runtime.ClientNotAvailableException: No activation for client sys.client/b6bc750f23ee452ead72a06eb3baddab) Response [S10.52.8.158:22222:110528910 sys.client/b6bc750f23ee452ead72a06eb3baddab+2bce487c693f45e2bbdf0e1a9f1af8dd]->[S10.52.8.158:22222:110528910 sys.svc.stream.agent/10.52.8.158:22222@110528910+MemProvider_5_memprovider-4-0x80000004] Orleans.Runtime.RejectionResponse #4712A6E508A615AE

[tomerpeled]

I'm also encountering the mentioned exception after upgrading to .net 9. On my end it seems that it happens only to grains with the ActivationCountBasedPlacement attribute in place. Is something was changed in regards to this placement?

[tomerpeled]

Any update?