Aaron Stannard

Possible causes / issues I want to investigate while we gather data from users: - [ ] `ShardRegion`s are out of sync - different nodes have different ideas on where...

When using `state-store-mode=ddata`, I think our built in replicator consistency settings are fine in v1.5: https://github.com/akkadotnet/akka.net/blob/f2867dfae4fd8c424b47c230c80f1921cffdd821/src/contrib/cluster/Akka.Cluster.Sharding/DDataShardCoordinator.cs#L116-L124 Can rule that out as a source of problems.

Might be a false alarm - first user had multiple clusters all writing to the same persistence store for v1.5 shard coordinator data.

```csharp akkaConfig.WithShardRegion( nameof(SchedulingManagerActor), _ => Props.Create(() => new SchedulingManagerActor(removeScheduleActor, scheduleMessageActor, projectionActor, builder.GetRequiredService()) ), new MessageExtractor(), new ShardOptions {StateStoreMode = StateStoreMode.DData, Role = "subscriber"}); ``` Ok, so no `remember-entities` and using...

So far, from the two users who have reported this issue to me (the third user had self-inflicted problems) - it looks like this issue occurs when state-store-mode = ddata...

 Spent a few hours going through this, looked at a few areas where a duplicate shard might be possible, but was able to rule them out. In...

I'm back from vacation and I'll be picking up work on this again

Yes! I have some data that indicates that this is a problem caused by a shard rebalancing / handoff timing out. Going to write a reproduction for that as soon...

These are the Akka.Remote settings for DotNetty that you're referring to? How many nodes are running in your cluster at any given time? That would determine whether or not these...

@brah-mcdude can you provide some actor code that we can look at in order to help reproduce?