DedicatedThreadPool Performance Optimization

[Aaronontheweb]

Version: 1.4.9

Prior to merging in https://github.com/akkadotnet/akka.net/pull/4511 , which moves all /system actors on a ForkJoinDispatcher, we need to go about improving the structure of the DedicatedThreadPool class (used by the ForkJoinDispatcher), which is sourced originally to this repository:

https://github.com/helios-io/DedicatedThreadPool

But that source code is referenced as a file directly into our project here:

https://github.com/akkadotnet/akka.net/blob/dev/src/core/Akka/Helios.Concurrency.DedicatedThreadPool.cs

The two code bases don't diverge much and that can easily be remedied. The last real change we made to the DedicatedThreadPool class was in 2017: https://github.com/akkadotnet/akka.net/pull/2714

Problems that #4511 May Introduce: Rise in Idle CPU Consumption

Running all of the /system actors on the ForkJoinDispatcher will help improve performance in really busy systems - it will allow workloads for Akka.Persistence, Akka.Cluster, etc independently from the primary .NET ThreadPool since the /system actors have to operate under real-time constraints. The only point of contention will be the OS thread scheduler, rather than the length of the work queue inside the shared ThreadPool. We've used this strategy successfully for years with Akka.Remote.

The problem is: what happens with ActorSytem instances that make virtually zero use of /system actors, such as the ActorSystems that are used inside stand-alone plugins like https://github.com/petabridge/Petabridge.Tracing.Zipkin ? The answer is we may start seeing increases in background CPU consumption as worker threads spin / wait / sleep for work, such as what we can observe with the HashedWheelTimerScheduler here: https://github.com/akkadotnet/akka.net/issues/4031

We need to avoid that to the extent that it's possible by freeing threads that don't have enough work do to and by adding new threads gradually as total demand increases. Spinning threads up and down is expensive, but having idle threads continuously sleeping and checking for work is also very expensive. We need to reach a happy medium.

Existing Issues with DedicatedThreadPool

Other issues that the DedicatedThreadPool has currently:

1. End-to-end throughput - it's at least 20% slower than the ThreadPool and possibly moreso now since the DTP hasn't benefited from any of the performance improvements introduced during subsequent versions of .NET Core. Finding ways to improve this through the queuing mechanism, thread activation, and work distribution would be a huge plus.
2. Thread pool scaling - I'm not confident that the current system is as effective as it should be when it comes to scaling thread pool usage relative to hardware. We should be able to scale the thread pool up and down relative to the number of virtual cores, unless explicitly overridden by configuration. Right now it seems that we only have the ability to explicitly configure the threadpool by computing what the desired number of threads should be externally. While we want to preserve the ability to explicitly limit the number of threads a DedicatedThreadPool can use, by default the DedicatedThreadPool should be able to dynamically adjust its thread count based on both demand and the number of virtual CPUs the machine has.

Constraints

1. Need to stick with .NET Standard 2.0 for now - adding dual targeting to Akka.NET is a substantial project and we are not prepared to do that yet.
2. All changes made to the DedicatedThreadPool need to have their own independent benchmarks, so it's probably best if we direct that work to the https://github.com/helios-io/DedicatedThreadPool repository (or fork it into this organization) and avoid running the giant Akka.NET test suite every time we make changes.
3. Need to achieve both: very low IDLE CPU footprint and better throughput.

[Zetanova]

I got already an issue with it https://github.com/akkadotnet/akka.net/issues/4636

@Aaronontheweb could u try my commit https://github.com/Zetanova/akka.net/tree/helios-idle-cpu I am somehow not capable to run the MultiNodeRunner

I try https://github.com/helios-io/DedicatedThreadPool out

[Aaronontheweb]

Made some massive progress towards replacing the DedicatedThreadPool here: https://github.com/akkadotnet/akka.net/pull/4882

This should get merged into v1.4.19 and deployed as an opt-in feature, with an area for collecting feedback from users here: https://github.com/akkadotnet/akka.net/discussions/4983

[Aaronontheweb]

Workaround

If you're running into CPU consumption issues today, you can always disable the internal-dispatcher (which uses the DedicatedThreadPool) using the following HOCON:

akka.actor.internal-dispatcher{
      type = "Dispatcher"
      executor = "default-executor"
}

This will have it run on the built-in .NET ThreadPool instead of the DedicatedThreadPool.

[Zetanova]

https://github.com/akkadotnet/akka.net/pull/7074 It is a PR to improve the default-executor and channel-executor

[Aaronontheweb]

Kind of solved this in v1.5.18 by just not using the DedicatedThreadPool.