IterationSetup for micro-benchmarks (`InvocationParam`)

[timcassell]

I remember seeing an issue about this, but I couldn't find it again.

Sometimes, in order to test a single responsibility, setup is necessary for each invocation. In cases where the benchmark method runs very quickly (micro-benchmarks), BDN complains about times less than 100ms due to inaccurate timings. Sometimes it's not possible to increase the number of operations while operating on the setup data (or doing so would affect the timing due to iterating over multiple setup datas).

Example:

[GenericTypeArguments(typeof(Vector4))]
[GenericTypeArguments(typeof(object))]
class AsyncBenchmark<T>
{
    TaskCompletionSource<T> _source;

    [IterationSetup]
    public void IterationSetup()
    {
        _source = new TaskCompletionSource<T>();
    }

    [Benchmark]
    public void AwaitPendingTask()
    {
        Run(_source.Task);
        _source.SetResult(default);

        static async void Run(Task<T> awaitable)
        {
            _ = await awaitable;
        }
    }
}

That could be changed to something like this.

[GenericTypeArguments(typeof(Vector4))]
[GenericTypeArguments(typeof(object))]
class AsyncBenchmark<T>
{
    const int invokeCount = 100_000;

    TaskCompletionSource<T>[] _sources = new TaskCompletionSource<T>[invokeCount];

    [IterationSetup]
    public void IterationSetup()
    {
        for (int i = 0; i < invokeCount; ++i)
        {
            _sources[i] = new TaskCompletionSource<T>();
        }
    }

    [Benchmark(OperationsPerInvoke = invokeCount)]
    public void AwaitPendingTask()
    {
        for (int i = 0; i < invokeCount; ++i)
        {
            var source = _sources[i];
            Run(source.Task);
            source.SetResult(default);
        }

        static async void Run(Task<T> awaitable)
        {
            _ = await awaitable;
        }
    }
}

Increasing invokeCount until the execution time >= 100ms. But that's a lot more code to write, it adds the overhead of iterating the array, it doesn't allow BDN to choose an optimal invocation count through heuristics, and the same code might still run faster than 100ms on a different machine.

So my idea is to have another attribute similar to ParamsSource that will update the value on each invocation.

[GenericTypeArguments(typeof(Vector4))]
[GenericTypeArguments(typeof(object))]
class AsyncBenchmark<T>
{
    [InvocationParamSource(nameof(CreateSource))]
    public TaskCompletionSource<T> Source { get; set; }
    public TaskCompletionSource<T> CreateSource() => new TaskCompletionSource<T>();

    [Benchmark]
    public void AwaitPendingTask()
    {
        Run(Source.Task);
        Source.SetResult(default);

        static async void Run(Task<T> awaitable)
        {
            _ = await awaitable;
        }
    }
}

The implementation of this could still use a storage array (or other collection), setting its length to the invocation count on each iteration, filling each index with the return value from CreateSource as part of the setup, then iterating over that collection and updating the value before invoking the benchmark method.

Iterating over the collection and updating the value (without calling CreateSource) could be measured as part the overhead and subtracted from the final result, giving us very accurate microbenchmarks. And we'll be able to write IterationSetup benchmarks that are portable to different machines. :)

[Edit] To support teardown:

[InvocationParam(nameof(CreateTarget), nameof(DisposeTarget))]
public MyDisposable Target { get; set; }
public MyDisposable CreateTarget() => new MyDisposable();
public void DisposeTarget(MyDisposable value) => value.Dispose();

[adamsitnik]

I've tried to implement InvocationSetup in the past: https://github.com/dotnet/BenchmarkDotNet/pull/1157

It was rejected for some good reasons. I agree that your proposal would make it work, but it would also make BDN ever more complex, and all our toolchains would need to support this (in-process, others + potentially source gen in the future). Since the need for that is quite rare and users can work around it like https://github.com/dotnet/performance/blob/10d532f792cdb147c00b97f3716c4bd842449d8f/src/benchmarks/micro/runtime/Span/Sorting.cs#L15 I am against adding it.

@timcassell Thank you for your time!

[timcassell]

@adamsitnik Thanks for the feedback. If things change in the future where this is wanted, I'd be willing to work on it for all toolchains, even after a source gen is added.

[timcassell]

I'm going to re-open this. The current IterationSetup strategy is confusing (we still see people asking about it all these years later), and inferior (with tiered jit, the benchmarked method might not reach tier1, along with reasons mentioned before).

No ETA on implementation, I just want to keep this on the radar.