NullReferenceException when activating stateless grain after upgrade to 3.6.2

[Nasicus]

We recently upgraded from 3.5.1 to 3.6.2. Since then we get from time to time this exception (note that it works sometimes!):

System.NullReferenceException: Object reference not set to an instance of an object.
   at ActivationData Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, bool newPlacement, string grainType, string genericArguments, Dictionary<string, object> requestContextData, out Task activatedPromise) in /_/src/Orleans.Runtime/Catalog/Catalog.cs:line 585
   at void Orleans.Runtime.Dispatcher.ReceiveMessage(Message message) in /_/src/Orleans.Runtime/Core/Dispatcher.cs:line 210

This grain is stateless and activated a lot of times. It looks like this:

    [StatelessWorker(maxLocalWorkers: 3)]
    [StorageProvider(ProviderName = "datastore")]
    internal class ReadOnlyProductGrain : Grain<ProductState>, IReadOnlyProductGrain
    {
        private Interfaces.ProductData.Product product;
        private long? brandId;
        private long? productTypeId;

        private readonly ILogger<ReadOnlyProductGrain> logger;

        public ReadOnlyProductGrain(ILogger<ReadOnlyProductGrain> logger)
        {
            this.logger = logger;
        }

        public override async Task OnActivateAsync()
        {
            try
            {
                if (State != null && !State.NeedsInitialization())
                {
                    product = new Interfaces.ProductData.Product
                    {
                        Availability = State.Availability,
                        Price = State.Price,
                        IsPublic = State.IsPublic,
                        IsOrderingAllowed = State.IsOrderingAllowed
                    };

                    brandId = State.Brand?.Id;
                    productTypeId = State.ProductType.Id;
                }
                else
                {
                    var productGrain = GrainFactory.GetGrain<IProductGrain>(this.GetPrimaryKeyString());
                    product = await productGrain.GetProduct();
                    brandId = await productGrain.GetBrandId();
                    productTypeId = await productGrain.GetProductTypeId();
                }
            }
            catch (Exception e)
            {
                logger.LogError(e, "Exception during grain activation");
                throw;
            }
        }

        public Task<long?> GetBrandId()
        {
            DeactivateOnIdle();
            return Task.FromResult(brandId);
        }

        public Task<long?> GetProductTypeId()
        {
            DeactivateOnIdle();
            return Task.FromResult(productTypeId);
        }

        public Task<Interfaces.ProductData.Price> GetPrice()
        {
            DeactivateOnIdle();
            return Task.FromResult(product.Price);
        }

        public Task<Interfaces.ProductData.Availability> GetAvailability()
        {
            DeactivateOnIdle();
            return Task.FromResult(product.Availability);
        }

        public Task<bool?> IsOrderingAllowed()
        {
            DeactivateOnIdle();
            return Task.FromResult(product.IsOrderingAllowed);
        }

        public Task<bool?> IsPublic()
        {
            DeactivateOnIdle();
            return Task.FromResult(product.IsPublic);
        }

        public Task<Interfaces.ProductData.Product> GetProduct()
        {
            DeactivateOnIdle();
            return Task.FromResult(product);
        }
    }
}

You may notice the many DeactivateOnIdle calls, because we don't want the Grain to stay activated / in memory.

So as a workaround we removed all this DeactivateOnIdle calls and instead added a very short time period when the graind should be deactivated:

	.Configure<GrainCollectionOptions>(options =>
	{
		options.ClassSpecificCollectionAge[typeof(ReadOnlyProductGrain).FullName!] =
			TimeSpan.FromSeconds(61);
	})

So bascially the question is: Is this a bug or are we doing something wrong? I.e. should we not call DeactivateOnIdle that often for a grain which is activated a lot of times?

[rafikiassumani-msft]

Triage: @Nasicus can you provide a GitHub repro project?

[ghost]

Hi @Nasicus We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[ghost]

Hi @Nasicus We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[Nasicus]

@rafikiassumani-msft unfortunately not, it's deeply integrated into one of our real projects.... if I have time I can try to reproduce it, but I'm not sure when that will be.

I was really mainly interested in knowing if the dev / you immediately could say "huh... yeah you're not supposed to call DeactivateOnIdle that often / in that manner" or something like that.

[ReubenBond]

Calling DeactivateOnIdle() ought to be fine. Is there any more to the stack trace than the 3 lines you posted?

[Nasicus]

I just checked and found another stack trace, a lot of it is from our own code though, so not sure if it's any help:

Pythia.Orleans.Grains.ProductList.ProductListImprovementException: Call to Product Grain 6332927 failed.
 Exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at ActivationData Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, bool newPlacement, string grainType, string genericArguments, Dictionary<string, object> requestContextData, out Task activatedPromise) in /_/src/Orleans.Runtime/Catalog/Catalog.cs:line 585
   at void Orleans.Runtime.Dispatcher.ReceiveMessage(Message message) in /_/src/Orleans.Runtime/Core/Dispatcher.cs:line 210
   at async Task<T> Orleans.Internal.OrleansTaskExtentions.ToTypedTask<T>(Task<object> task)+ConvertAsync(?) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 114
   at async Task<Product> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.GetProduct(long productId, int portalId) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 148
   at async Task<bool> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.PriceDropCheck(ProductListItem entry, int portalId) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 133
   at void Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain+<>c__DisplayClass12_0+<<SelectProductListItemForNotification>b__7>d.MoveNext() in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 158
   at void Pythia.Shared.AsyncUtils+<>c__DisplayClass0_0<TInput, TOutput>+<<RestrictedWhenAll>g__Apply|3>d.MoveNext() in /app/Pythia.Shared/AsyncUtils.cs:line 47;Call to Product Grain 6881946 failed.
 Exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at ActivationData Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, bool newPlacement, string grainType, string genericArguments, Dictionary<string, object> requestContextData, out Task activatedPromise) in /_/src/Orleans.Runtime/Catalog/Catalog.cs:line 585
   at void Orleans.Runtime.Dispatcher.ReceiveMessage(Message message) in /_/src/Orleans.Runtime/Core/Dispatcher.cs:line 210
   at async Task<T> Orleans.Internal.OrleansTaskExtentions.ToTypedTask<T>(Task<object> task)+ConvertAsync(?) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 114
   at async Task<Product> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.GetProduct(long productId, int portalId) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 148
   at async Task<bool> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.PriceDropCheck(ProductListItem entry, int portalId) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 133
   at void Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain+<>c__DisplayClass12_0+<<SelectProductListItemForNotification>b__7>d.MoveNext() in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 158
   at void Pythia.Shared.AsyncUtils+<>c__DisplayClass0_0<TInput, TOutput>+<<RestrictedWhenAll>g__Apply|3>d.MoveNext() in /app/Pythia.Shared/AsyncUtils.cs:line 47
   at async Task<ProductListItem> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.SelectProductListItemForNotification(int portalId, IReadOnlyList<ProductListItem> productListEntries, Func<ProductListItem, int, Task<bool>> validityCheckFn, ISet<long> exclusions, Func<ProductListItem, int> orderingByFn) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 198
   at async Task<ProductListItem> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.GetGreatestPriceDrop(int portalId, IReadOnlyList<ProductListItem> productListEntries, IProductListImprovementNotificationGrain productListImprovementNotificationGrain) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 110
   at async Task Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.ReceiveReminder(string reminderName, TickStatus status) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 97
   at async Task<object> Orleans.OrleansCodeGenRemindableMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request) in /_/src/Orleans.Core/obj/Release/net6.0/Orleans.Core.orleans.g.cs:line 5215
   at async Task Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
   at async Task OrleansDashboard.Metrics.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)
   at async Task Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
   at async Task Orleans.Runtime.InsideRuntimeClient.Invoke(IAddressable target, IInvokable invokable, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 469
   at async Task Orleans.Runtime.ReminderService.LocalReminderService+LocalReminderData.OnTimerTick() in /_/src/Orleans.Runtime/ReminderService/LocalReminderService.cs:line 631

[ReubenBond]

@Nasicus apologies for the slow response. Are you able to provide a memory dump? If you are running on Windows, you can use procdump to capture a dump as the NullReferenceException is thrown: procdump -e 1 -f "NullReferenceException" YourApp.exe

[ReubenBond]

Fix PR: #7918 Apologies for the hassle, @Nasicus!