Async GlobalSetup does not work with InProcess toolchain

[mstefarov]

trafficstars

https://github.com/dotnet/BenchmarkDotNet/issues/521 added support for async GlobalSetup, but it appears that InProcess toolchain does not wait for the async setup method to finish. Simple reproducer:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Threading.Tasks;

namespace AsyncGlobalSetupRepro
{
    [InProcess]
    public class MyBenchmarkClass
    {
        private string _foo;

        [GlobalSetup]
        public async Task MyAsyncSetup()
        {
            await Task.Delay(1000);
            _foo = "hello";
        }

        [Benchmark]
        public int MyBenchmark() => _foo.Length; // NullReferenceException!
    }

    public class Program
    {
        public static void Main(string[] args)
        {
            BenchmarkRunner.Run(typeof(MyBenchmarkClass));
        }
    }
}

It looks like MyAsyncSetup gets invoked as a regular method here, and then JITting starts immediately without waiting for it to finish: https://github.com/dotnet/BenchmarkDotNet/blob/e7ff4cefcc8d429205a21a76045e82688ee063c5/src/BenchmarkDotNet/Engines/EngineFactory.cs#L28

[corrieriluca]

Hi, I experience the same problem here ! ✋

When using BDN with dotnet/crank, the --inProcess flag is passed to the BDN executable, and my benchmark throws a NullReferenceException. By testing my benchmarks directly with BDN (not through Crank) I realized that my async GlobalSetup method was not awaited/executed when the --inProcess flag is toggled.

[Turnerj]

Yeah, just hit this myself. Looks like #1968 will fix this so hopefully that gets merged in soon.

[Rambolink94]

Are there any updates on the progress of this? I ran into this issue today with InProcess in an xunit test.

[timcassell]

I've had an open PR to fix this for a while, it just depends on when @AndreyAkinshin will get around to reviewing it.

In the meantime, you can work around this issue by synchronously waiting on your task task.GetAwaiter().GetResult().

[agra6475]

Encountered this bug well over a year ago and found this issue. Returned to it today to see if it was fixed. Quite surprised that this issue is still open if all that's needed is for someone to review the fix.

Is there is a reason it hasn't been merged yet?