[Docs] [Question] Resolving split-brain scenario

[peter-perot]

Almost all code examples that involve stateful grains write data to the storage but avoid reading them back programmatically. The reason for this practice is that Orleans automatically reads the state on activation, and when there are no other "leaks" that change the state on the underlying database (e.g. as done by stored procedures), there is no reason to read the data back once it is cached in memory by the grain.

But now consider a split-brain scenario, where grain A reads the state at time T1, and a duplicate activation A' of the same grain (due to split-brain) gets an update call the state of which is written to the storage at time T2 > T1.

Now Orleans detects the split-brain scenario and tries to resolve it. But how? I think we need a more detailed documentation here how Orleans achieves this.

Strategy 1: If Orleans only deactivates grain A', then grain A with a stale state stays in memory, and future read-requests will all return stale data.

Strategy 2: If Orleans deactivates both grains, then the next read-request will reinstantiate the grain and re-read the latest state, so we get the correct state in this case.

Q1: What is the strategy that is implemented in Orleans?

Q2: In case the first strategy is implemented, what are the recommended countermeasures to avoid staleness? Do we need to read the state from storage on each read-request? [Clarification: Interim staleness can be tolerated, but staleness forever (in case the state is not changed again) is no option.]

There are two options for Q2 that come into my mind:

Option 1: Re-read the state when a read-request comes in and the last refresh from the storage is older than a defined period.

Option 2: Re-read the state when a read-request comes in and the last refresh from the storage is older than a defined period, but don't suspend the current call until the re-read has finished. Instead return the current potentially stale state and trigger a refresh in the background (with a non-awaited task).