rewrite `HashedWheelTimer` to use `await` instead of `Thread.Sleep`

[Aaronontheweb]

close #4031

An experiment for the time being - looking to see how it performs in benchmarks and in the test suite.

Ideally, need to refactor this so it runs on the /system dispatcher but that will require changing the constructor arguments.

Changes

Removes the dedicated Thread from the HashedWheelTimerScheduler and relies on ValueTask and await instead.

Checklist

For significant changes, please ensure that the following have been completed (delete if not relevant):

• [x] This change follows the Akka.NET API Compatibility Guidelines.
• [x] This change follows the Akka.NET Wire Compatibility Guidelines.
• [ ] I have reviewed my own pull request.
• [ ] Design discussion issue #4031
• [ ] Changes in public API reviewed, if any.

Latest dev Benchmarks

WIP

This PR's Benchmarks

WIP

[Aaronontheweb]

Just ran a simple test so far - formed a cluster with this change no problem. Curious to see if the test suite explodes on contact or not.

[Aaronontheweb]

If this escapes beta stage, I'm going to fork the HashedWheelTimer and include a copy that runs with the old behavior and set that to default in order to limit side effects on new users. Users with CPU utilization concerns can change their settings in order to take advantage.

[Aaronontheweb]

Looks to me like this change definitely doesn't quite cut the mustard for some of the racy specs here - I'll need to harden the implementation some.

[Aaronontheweb]

RemotePingPong

dev

OSVersion: Microsoft Windows NT 10.0.19041.0
ProcessorCount: 16
ClockSpeed: 0 MHZ
Actor Count: 32
Messages sent/received per client: 200000 (2e5)
Is Server GC: True
Thread count: 112

Num clients, Total [msg], Msgs/sec, Total [ms], Start Threads, End Threads
1, 200000, 159617, 1253.23, 112, 140
5, 1000000, 320103, 3124.93, 148, 161
10, 2000000, 335009, 5970.23, 169, 169
15, 3000000, 316189, 9488.85, 177, 162
20, 4000000, 193733, 20647.86, 170, 141
25, 5000000, 325733, 15350.78, 150, 142
30, 6000000, 327583, 18316.31, 150, 138

This PR

OSVersion: Microsoft Windows NT 10.0.19041.0 ProcessorCount: 16 ClockSpeed: 0 MHZ Actor Count: 32 Messages sent/received per client: 200000 (2e5) Is Server GC: True Thread count: 112

Num clients, Total [msg], Msgs/sec, Total [ms], Start Threads, End Threads 1, 200000, 156863, 1275.02, 112, 153 5, 1000000, 319082, 3134.05, 161, 161 10, 2000000, 328840, 6082.80, 169, 169 15, 3000000, 313907, 9557.00, 177, 162 20, 4000000, 255331, 15666.90, 170, 140 25, 5000000, 201078, 24866.44, 148, 139 30, 6000000, 317243, 18913.73, 147, 139

Needed to make sure that the performance and thread counts were roughly the same as this change shouldn't have any impact there other than reducing total thread count by 1, roughly.

[Aaronontheweb]

No test failures, but several parts of the test suite timed out.

edit: makes me wonder if there's an "unclean shutdown" problem with the new scheduler as-is.

[Aaronontheweb]

actually, seeing quite a few failures in the Akka.Streams testsuite on my local machine

[Aaronontheweb]

I'll give this a go - otherwise I'll give wait handles a try.

[Aaronontheweb]

Superseded by https://github.com/akkadotnet/akka.net/pull/6435