ArgumentsSource method is executed before GlobalSetup method

[drieseng]

In BenchmarkDotNet version 0.12, the method that provided arguments for a parameterized benchmark is executed before the GlobalSetup method of the benchmark.

Having the designated GlobalSetup method execute before the ArgumentsSource method is useful when the arguments are costly to create, and - for example - need to be disposed once the benchmark has completed.

Example:

using BenchmarkDotNet.Attributes;
using System;
using System.Collections.Generic;
using System.IO;

public class ArgumentsSourceBenchmarks
{
    private List<DownloadFile> _downloadFiles;
    private StreamProvider _streamProvider;

    [GlobalSetup]
    public void GlobalSetup()
    {
        _streamProvider = new StreamProvider();
        _downloadFiles = new List<DownloadFile>
            {
                new DownloadFile("A", _streamProvider.Create(2048)),
                new DownloadFile("B", _streamProvider.Create(4096))
            };
    }

    [GlobalCleanup]
    public void GlobalCleanup()
    {
        _streamProvider?.Dispose();

        foreach (var downloadFile in _downloadFiles)
        {
            downloadFile.DownloadStream.Dispose();
        }
    }

    [Benchmark]
    [ArgumentsSource(nameof(DownloadFileArguments))]
    public void DownloadFileBenchmark(string path, Stream stream)
    {
        // Do something
    }

    public IEnumerable<object[]> DownloadFileArguments()
    {
        foreach (var downloadFile in _downloadFiles)
        {
            yield return new object[] { downloadFile.Path, downloadFile.DownloadStream };
        }
    }

    public class DownloadFile
    {
        public DownloadFile(string path, Stream downloadStream)
        {
            Path = path;
            DownloadStream = downloadStream;
        }

        public string Path { get; }
        public Stream DownloadStream { get; }
    }

    public class StreamProvider : IDisposable
    {
        public void Dispose()
        {
        }

        public Stream Create(int initialCapacity)
        {
            return new MemoryStream(initialCapacity);
        }
    }
}

Running this benchmark class will current result in an NRE:

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at ArgumentsSourceBenchmarks.<DownloadFileArguments>d__4.MoveNext() in D:\Development\Repos\ArgumentsSource\ArgumentsSourceBenchmarks.cs:line 32
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkConverter.GetValidValuesForParamsSource(Type parentType, String sourceName)
   at BenchmarkDotNet.Running.BenchmarkConverter.<GetArgumentsDefinitions>d__9.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkConverter.MethodsToBenchmarksWithFullConfig(Type containingType, MethodInfo[] benchmarkMethods, ImmutableConfig immutableConfig)
   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
   at Program.Main(String[] args) in D:\Development\Repos\ArgumentsSource\Program.cs:line 13

[adamsitnik]

Hi @drieseng

Thank you for a very detailed bug report.

I agree that the problem exists and should be solved.

For a now, I can offer a workaround that I've been using myself so far:

moving the setup logic to the ctor of the type returned by [ArgumentsSource]

You can find the example here.

What is important is that you can override the .ToString method in given type to change the way the arguments are represented in the results. Example: https://github.com/dotnet/performance/blob/2a4232d994b88c37093399537b0b2a62ef163aeb/src/benchmarks/micro/libraries/System.Text.Encodings.Web/Perf.Encoders.cs#L74

[ValeriyGourov]

For a now, I can offer a workaround that I've been using myself so far:

moving the setup logic to the ctor of the type returned by [ArgumentsSource]

You can find the example here.

Hi @adamsitnik

How can I initialize my data for ArgumentsSource asynchronously? For example:

using System.Diagnostics;

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;

IConfig config = Debugger.IsAttached
    ? new DebugInProcessConfig()
    : DefaultConfig.Instance;

_ = BenchmarkRunner.Run<Benchmarks>(config);

public class Benchmarks
{
    private List<SomeClass> _data = [];

    [GlobalSetup(Target = nameof(Test1))]
    public async Task Test1GlobalSetup()
    {
        await Task.Delay(100);

        _data = [ new (), new () ];
    }

    public IEnumerable<SomeClass> TestData => _data;

    [Benchmark]
    [ArgumentsSource(nameof(TestData))]
    public Task Test1(SomeClass someClass) => Task.CompletedTask;
}

public class SomeClass
{
    public override string ToString() => nameof(SomeClass);
}

But it doesn't work.