Aaron Stannard

So this bug likely affected less people than I initially thought as https://github.com/akkadotnet/akka.net/blob/d1ed226e8b140215427bbd8ffd58130662d7ff28/src/contrib/cluster/Akka.Cluster.Tools/Singleton/reference.conf#L49 Has been set to `false` this whole time and that's also the default value from the HOCON...

Looks like the original issue reported by the end user wasn't even caused by the `AppVersion`, but this feature is definitely a footgun and probably needs to be removed.

A bunch of these default timeouts are configurable in the testkit, but yes those two are not. We run into these same issues inside the Akka.NET test suite itself and...

Thank you for the reproduction! I'll see what we can do here.

So I tried this out myself and was able to get the application to process everything in about 3:30, which is still too slow IMHO: https://share.linqpad.net/qa5jrmw9.linq Trick was to make...

I think the problem is here: https://github.com/akkadotnet/akka.net/blob/1af82a75c58d943f0fe802b566dbfe8b78e34203/src/core/Akka.Streams/Dsl/Hub.cs#L940-L948 The code we use for determining which consumers get which message looks complicated to me - and based on my profiling data we're...

The performance issues have nothing to do with boxing or any measurable CPU issues - it's a scheduling problem. There's long periods of time where the consumers aren't doing anything....

> I started trying to adjust parameters, mainly distributed-data.max-delta-elements, but it had no effect. I suspect that max-delta-elements is used to split multiple IReplicatedData, while the entire ORSet is considered...

Looks like we did address a bug related to this in https://github.com/akkadotnet/akka.net/issues/5157 - is it possible that the elements in your `ORSet` are too large to fit into the frame...

Nice catch! Yes, we should apply your fix and just re-do the case labels here. If you'd like to send in a small PR with a reproduction spec we'd be...