Question: custom persistence of IGrainObserver

[DunetsNM]

As per title, is it possible to explicitly persist and restore IGrainObserver using a custom storage / bypassing Orleans storage interfaces?

All I was able to find is a closed question:

https://github.com/dotnet/orleans/issues/7371

It says

Can we use IPersistentState to store IGrainObserver subscibed to a grain? Yes you can.

but doesn't elaborate on details

Anyways, I would prefer to not use IPersistentState, because we have our own explicit save to storage on a grain call (SQL in a transaction), and would like to persist observers too in the same SQL transaction. And in OnActivate we want to restore the persisted observers.

[ReubenBond]

When the observer represents an object reference (anything created by calling IGrainFactory.CreateObserverReference), you should not do this because clients have random identities, which change every time they are started. If you know that your observers are all actually grains, then it's fine to do this. Technically, it will work for object references, but if the client process has restarted, then you will get a ClientNotAvailableException when you try to call the reference (make sure that your grain observer methods return Task or another non-void type so that you can observe the exception).

The easiest way to store a grain/observer reference in Orleans without going through Orleans' own serialization system is to call myReference.GetGrainId(), which returns a GrainId struct. From there, you can either serialize it with ToString() and use GrainId.TryParse to parse it, or you can serialize the individual Type and Key components. Once you want to get a reference back, you can use IGrainFactory.GetGrain<TInterface>(GrainId) to create the original grain reference. GrainIds are formatted as type/key, eg shopping-cart/dunetsnm.

Here's some code for the process:

// Create a grain reference and convert its id into a string
var myGrain = grainFactory.GetGrain<IMyGrain>("test");
var grainId = myGrain.GetGrainId();
var grainIdString = grainId.ToString(); // "mygrain/test"

// ... store and load the grain id ...

// Parse the grain id string back into a grain reference
GrainId.TryParse(grainIdString, our var rehydratedId);
var rehydratedGrainReference = grainFactory.GetGrain<IMyGrain>(rehydratedId);

[DunetsNM]

thanks for your reply

When the observer represents an object reference (anything created by calling IGrainFactory.CreateObserverReference), you should not do this because clients have random identities, which change every time they are started

do you mean IGrainFactory.CreateObjectReference ? (I use Orleans 3.x). That's exactly what I do though, my observer is on the client side / not in a silo.

However in my scenario observer is short living and only awaits for a single notification that is usually expected to be delivered within 1 to 5 seconds but can take longer. Client awaits for observer notification or 30-40 sec timeout, whatever comes first, then grain observer can be garbage collected. Also it's not a problem if client shuts down before receiving the notification, restarted client will not care about that.

In fact I would be satisfied with either of two outcomes:

1. non-persistent state + fail fast: When the silo shuts down or grain gets recycled and loses the reference to IGrainObserver, client becomes aware of it somehow and can stop awaiting for notification that will never happen (ideally including not graceful silo shutdown). I have some hacky ideas of workaround e.g. keep polling the grain to see if it was recycled but maybe there's a better way out of the box.

2. persistent state + fail slow: if silo shuts down grain will get recycled it can restore the observer and still deliver the notification before timeout

Currently I have worst of two: non-persistent state + fail slow: silo shuts down and loses reference to IGrainObserver forever but client is unaware and keeps waiting for the full 30-40 seconds timeout.

[ReubenBond]

Yes, I meant IGrainFactory.CreateObjectReference - thanks for catching that. As for your options, you could consider sending a notification to the clients when your grain is being deactivated, to tell them not to expect any further replies.

[DunetsNM]

sending a notification to the clients when your grain is being deactivated

right, this will not work for aborted silo process though (is there any way to know about it at all?), but still better than nothing

[slawomirpiotrowski]

You can implement ISiloLifecycleObserver (and register it using ILifecycleParticipant<IClusterClientLifecycle> service) to be notified whenever any silo gets terminated (method SiloTerminated of lifecycle observer will be called). If your grain was placed on a terminated silo then that grain was also terminated, and it might be without calling shutdown.