orleans
orleans copied to clipboard
dotnet
NullReferenceException when activating stateless grain after upgrade to 3.6.2
We recently upgraded from 3.5.1 to 3.6.2.
Since then we get from time to time this exception (note that it works sometimes!):
System.NullReferenceException: Object reference not set to an instance of an object.
at ActivationData Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, bool newPlacement, string grainType, string genericArguments, Dictionary<string, object> requestContextData, out Task activatedPromise) in /_/src/Orleans.Runtime/Catalog/Catalog.cs:line 585
at void Orleans.Runtime.Dispatcher.ReceiveMessage(Message message) in /_/src/Orleans.Runtime/Core/Dispatcher.cs:line 210
This grain is stateless and activated a lot of times.
It looks like this:
[StatelessWorker(maxLocalWorkers: 3)]
[StorageProvider(ProviderName = "datastore")]
internal class ReadOnlyProductGrain : Grain<ProductState>, IReadOnlyProductGrain
{
private Interfaces.ProductData.Product product;
private long? brandId;
private long? productTypeId;
private readonly ILogger<ReadOnlyProductGrain> logger;
public ReadOnlyProductGrain(ILogger<ReadOnlyProductGrain> logger)
{
this.logger = logger;
}
public override async Task OnActivateAsync()
{
try
{
if (State != null && !State.NeedsInitialization())
{
product = new Interfaces.ProductData.Product
{
Availability = State.Availability,
Price = State.Price,
IsPublic = State.IsPublic,
IsOrderingAllowed = State.IsOrderingAllowed
};
brandId = State.Brand?.Id;
productTypeId = State.ProductType.Id;
}
else
{
var productGrain = GrainFactory.GetGrain<IProductGrain>(this.GetPrimaryKeyString());
product = await productGrain.GetProduct();
brandId = await productGrain.GetBrandId();
productTypeId = await productGrain.GetProductTypeId();
}
}
catch (Exception e)
{
logger.LogError(e, "Exception during grain activation");
throw;
}
}
public Task<long?> GetBrandId()
{
DeactivateOnIdle();
return Task.FromResult(brandId);
}
public Task<long?> GetProductTypeId()
{
DeactivateOnIdle();
return Task.FromResult(productTypeId);
}
public Task<Interfaces.ProductData.Price> GetPrice()
{
DeactivateOnIdle();
return Task.FromResult(product.Price);
}
public Task<Interfaces.ProductData.Availability> GetAvailability()
{
DeactivateOnIdle();
return Task.FromResult(product.Availability);
}
public Task<bool?> IsOrderingAllowed()
{
DeactivateOnIdle();
return Task.FromResult(product.IsOrderingAllowed);
}
public Task<bool?> IsPublic()
{
DeactivateOnIdle();
return Task.FromResult(product.IsPublic);
}
public Task<Interfaces.ProductData.Product> GetProduct()
{
DeactivateOnIdle();
return Task.FromResult(product);
}
}
}
You may notice the many DeactivateOnIdle calls, because we don't want the Grain to stay activated / in memory.
So as a workaround we removed all this DeactivateOnIdle calls and instead added a very short time period when the graind should be deactivated:
.Configure<GrainCollectionOptions>(options =>
{
options.ClassSpecificCollectionAge[typeof(ReadOnlyProductGrain).FullName!] =
TimeSpan.FromSeconds(61);
})
So bascially the question is:
Is this a bug or are we doing something wrong? I.e. should we not call DeactivateOnIdle that often for a grain which is activated a lot of times?
Hi @Nasicus We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.
Hi @Nasicus We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.
@rafikiassumani-msft unfortunately not, it's deeply integrated into one of our real projects.... if I have time I can try to reproduce it, but I'm not sure when that will be.
I was really mainly interested in knowing if the dev / you immediately could say "huh... yeah you're not supposed to call DeactivateOnIdle that often / in that manner" or something like that.
Calling DeactivateOnIdle() ought to be fine. Is there any more to the stack trace than the 3 lines you posted?
I just checked and found another stack trace, a lot of it is from our own code though, so not sure if it's any help:
Pythia.Orleans.Grains.ProductList.ProductListImprovementException: Call to Product Grain 6332927 failed.
Exception:
System.NullReferenceException: Object reference not set to an instance of an object.
at ActivationData Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, bool newPlacement, string grainType, string genericArguments, Dictionary<string, object> requestContextData, out Task activatedPromise) in /_/src/Orleans.Runtime/Catalog/Catalog.cs:line 585
at void Orleans.Runtime.Dispatcher.ReceiveMessage(Message message) in /_/src/Orleans.Runtime/Core/Dispatcher.cs:line 210
at async Task<T> Orleans.Internal.OrleansTaskExtentions.ToTypedTask<T>(Task<object> task)+ConvertAsync(?) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 114
at async Task<Product> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.GetProduct(long productId, int portalId) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 148
at async Task<bool> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.PriceDropCheck(ProductListItem entry, int portalId) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 133
at void Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain+<>c__DisplayClass12_0+<<SelectProductListItemForNotification>b__7>d.MoveNext() in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 158
at void Pythia.Shared.AsyncUtils+<>c__DisplayClass0_0<TInput, TOutput>+<<RestrictedWhenAll>g__Apply|3>d.MoveNext() in /app/Pythia.Shared/AsyncUtils.cs:line 47;Call to Product Grain 6881946 failed.
Exception:
System.NullReferenceException: Object reference not set to an instance of an object.
at ActivationData Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, bool newPlacement, string grainType, string genericArguments, Dictionary<string, object> requestContextData, out Task activatedPromise) in /_/src/Orleans.Runtime/Catalog/Catalog.cs:line 585
at void Orleans.Runtime.Dispatcher.ReceiveMessage(Message message) in /_/src/Orleans.Runtime/Core/Dispatcher.cs:line 210
at async Task<T> Orleans.Internal.OrleansTaskExtentions.ToTypedTask<T>(Task<object> task)+ConvertAsync(?) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 114
at async Task<Product> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.GetProduct(long productId, int portalId) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 148
at async Task<bool> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.PriceDropCheck(ProductListItem entry, int portalId) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 133
at void Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain+<>c__DisplayClass12_0+<<SelectProductListItemForNotification>b__7>d.MoveNext() in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 158
at void Pythia.Shared.AsyncUtils+<>c__DisplayClass0_0<TInput, TOutput>+<<RestrictedWhenAll>g__Apply|3>d.MoveNext() in /app/Pythia.Shared/AsyncUtils.cs:line 47
at async Task<ProductListItem> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.SelectProductListItemForNotification(int portalId, IReadOnlyList<ProductListItem> productListEntries, Func<ProductListItem, int, Task<bool>> validityCheckFn, ISet<long> exclusions, Func<ProductListItem, int> orderingByFn) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 198
at async Task<ProductListItem> Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.GetGreatestPriceDrop(int portalId, IReadOnlyList<ProductListItem> productListEntries, IProductListImprovementNotificationGrain productListImprovementNotificationGrain) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 110
at async Task Pythia.Orleans.Grains.ProductList.ProductListImprovementGrain.ReceiveReminder(string reminderName, TickStatus status) in /app/Pythia.Orleans.Grains/ProductList/ProductListImprovementGrain.cs:line 97
at async Task<object> Orleans.OrleansCodeGenRemindableMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request) in /_/src/Orleans.Core/obj/Release/net6.0/Orleans.Core.orleans.g.cs:line 5215
at async Task Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at async Task OrleansDashboard.Metrics.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)
at async Task Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at async Task Orleans.Runtime.InsideRuntimeClient.Invoke(IAddressable target, IInvokable invokable, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 469
at async Task Orleans.Runtime.ReminderService.LocalReminderService+LocalReminderData.OnTimerTick() in /_/src/Orleans.Runtime/ReminderService/LocalReminderService.cs:line 631
@Nasicus apologies for the slow response. Are you able to provide a memory dump? If you are running on Windows, you can use procdump to capture a dump as the NullReferenceException is thrown: procdump -e 1 -f "NullReferenceException" YourApp.exe