BenchmarkDotNet Check BenchmarkRunner arguments

It affects BenchmarkRunner.Run* with args==null only.

The first commit is not breaking change. It fast exits with message when no benchmarks to execute or when the benchmark crushes (when arguments are null).

The second commit adds fast exit when MethodInfo or Types are not benchmarkable.

Types

//Master
BenchmarkSwitcher.FromTypes(new Type[] { }).Run(); // message and fast exit
BenchmarkRunner.Run(new Type[] { }, args: new[] { "-f", "*" } ); // too because it switches to BenchmarkSwitcher
BenchmarkRunner.Run(new Type[] { }); //silently trying execute. There is no benchmark methods so it returns 0

Now, the last line shows message too.

MethodInfo

In master, code below silently filters methods, now ALL methods should be benchmarkable and being of BenchmarkClass.

BenchmarkRunner.Run(typeof(BenchmarkClass), new MethodInfo[] { 
    BenchmarkClass.CorrectMethod,
    BenchmarkClass.PrivateMethod, //wrong
    string.Concat, //wrong
    OtherBenchmarkClass.CorrectMethod  }); //wrong

closes #1899 closes #1983

Feb 02 '22 20:02 YegorStepanov

While you're at it, there is also this:

Affects:
public static class BenchmarkConverter {
    public static BenchmarkRunInfo MethodsToBenchmarks(Type containingType, MethodInfo[] benchmarkMethods, IConfig config = null) {}
}
public static class BenchmarkRunner {
    public static Summary Run(Type type, MethodInfo[] methods, IConfig config = null) {}
} 
BDN doesn't check if the given MethodInfos belong to the given Type and will convert the methods anyway.

In case of a mismatch, the DisplayInfo will be wrong because BDN combines the class name from Type with the method name from MethodInfo despite the latter having a different ReflectedType (DeclaringType can be different if base class).

For out-of-process jobs, building the auto-generated boilerplate code fails, because the Runnable_0 etc. classes are derived from the wrong type.

In-process jobs are executed w/o failing, but display the wrong info.

I made a note of it a while ago with the intent to fix it myself, but haven't gotten around to it yet. Thought I mention it, since you're working in that area...

No pressure, though 😉

Feb 02 '22 23:02 mawosoft

@mawosoft Sorry for stealing the issue :)

BDN doesn't check if the given MethodInfos belong to the given Type and will convert the methods anyway.

Сheck was added.

For out-of-process jobs, building the auto-generated boilerplate code fails, because the Runnable_0 etc. classes are derived from the wrong type.

In-process jobs are executed w/o failing, but display the wrong info.

If I understand you correctly, does this apply to this code?

BenchmarkRunner.Run(typeof(BenchmarkClass), new MethodInfo[] { 
    BenchmarkClass.CorrectMethod,
    BenchmarkClass.PrivateMethod, 
    string.Concat, 
    OtherBenchmarkClass.CorrectMethod  });

I though it just filtering somewhere in BenchmarkRunnerClean.Run().

For reviewers:

We can't add checks to BenchmarkConverter.TypeToBenchmarks(). For example, if we added null check for type argument, some tests would fail (smth related to new BenchmarkSwitcher(*Filter*))

~~For consistency BenchmarkConverter.MethodsToBenchmarks() is dangerous too.~~

Oh, after today's commit by Adam it became much more safe.

Feb 02 '22 23:02 YegorStepanov

Сheck was added.

My bad. I only skimmed through it and didn't realize you've already done it.

For out-of-process jobs, building the auto-generated boilerplate code fails, because the Runnable_0 etc. classes are derived from the wrong type.

In-process jobs are executed w/o failing, but display the wrong info.

If I understand you correctly, does this apply to this code?
BenchmarkRunner.Run(typeof(BenchmarkClass), new MethodInfo[] { 
    BenchmarkClass.CorrectMethod,
    BenchmarkClass.PrivateMethod, 
    string.Concat, 
    OtherBenchmarkClass.CorrectMethod  });

Yeah, particularly the last one. The other two were skipped I think (I don't exactly remember the finer details).

Feb 03 '22 00:02 mawosoft

Hi Adam, no problems, I will gladly take the time for this.

Aug 25 '22 11:08 YegorStepanov

Code

public class Program
{
    static int benchmarkNumber = 0;
    static string[] ARGS = null;
    static string[] ARGS = Array.Empty<string>();

    static void Main()
    {
        Console.WriteLine("null:");
        RunType(null as Type);
        RunTypes(null as Type[]);
        RunAssembly(null as Assembly);
        RunInfo(null as BenchmarkRunInfo);
        RunInfos(null as BenchmarkRunInfo[]);
        RunTypeMethods(null as Type, null as MethodInfo[]);

        Console.WriteLine("RunT");
        RunT<string>(DefaultConfig.Instance);
        RunT<string>();
        RunT<SealedBenchmarkClass>();

        Console.WriteLine("RunTypes");
        RunTypes(Array.Empty<Type>());
        RunTypes(new Type[] { typeof(string) });
        RunTypes(new Type[] { typeof(string), typeof(Benchmark1), typeof(object), typeof(List<string>) });
        RunTypes(new Type[] { typeof(string), null });
        RunTypes(new Type[] { null, typeof(Benchmark1) });

        Console.WriteLine("RunMethods");
        MethodInfo strMethod = typeof(string).GetMethods().First(m => m.Name == "Remove");
        MethodInfo benchmarkMethod = typeof(Benchmark1).GetMethod("M1");

        RunTypeMethods(typeof(Benchmark1), new MethodInfo[] { benchmarkMethod, null });
        RunTypeMethods(typeof(Benchmark1), new MethodInfo[] { strMethod });
        RunTypeMethods(typeof(string), new MethodInfo[] { benchmarkMethod });

        RunTypeMethods(typeof(string), new MethodInfo[] { });
        RunTypeMethods(typeof(Benchmark1), new MethodInfo[] { });

        RunTypeMethods(typeof(Benchmark1), new MethodInfo[] { benchmarkMethod, typeof(Benchmark2).GetMethod("M1") });
        RunTypeMethods(typeof(Benchmark1), typeof(Benchmark2).GetMethods());

        Console.WriteLine("RunAssembly");
        RunAssembly(typeof(string).Assembly);

        Console.WriteLine("RunInfo");
        var config = DefaultConfig.Instance.CreateImmutableConfig();

        RunInfo(new BenchmarkRunInfo(null, null, null));
        RunInfo(new BenchmarkRunInfo(null, null, config));

        RunInfo(new BenchmarkRunInfo(null, typeof(Benchmark1), null));
        RunInfo(new BenchmarkRunInfo(null, typeof(Benchmark1), config));

        RunInfo(new BenchmarkRunInfo(Array.Empty<BenchmarkCase>(), null, null));
        RunInfo(new BenchmarkRunInfo(Array.Empty<BenchmarkCase>(), null, config));

        RunInfo(new BenchmarkRunInfo(Array.Empty<BenchmarkCase>(), typeof(Benchmark1), null));
        RunInfo(new BenchmarkRunInfo(Array.Empty<BenchmarkCase>(), typeof(Benchmark1), config));
    }

    static void RunT<T>(IConfig config = null) =>
        Catch(() => BenchmarkRunner.Run<T>(config, args: ARGS));

    static void RunType(Type type) =>
        Catch(() => BenchmarkRunner.Run(type, args: ARGS));

    static void RunTypes(Type[] types) =>
        Catch(() => BenchmarkRunner.Run(types, args: ARGS));

    static void RunTypeMethods(Type type, MethodInfo[] methods) =>
        Catch(() => BenchmarkRunner.Run(type, methods)); //no args overload

    static void RunAssembly(Assembly assembly) =>
        Catch(() => BenchmarkRunner.Run(assembly, args: ARGS));

    static void RunInfo(BenchmarkRunInfo info) =>
        Catch(() => BenchmarkRunner.Run(info)); //no args overload

    static void RunInfos(BenchmarkRunInfo[] infos) =>
        Catch(() => BenchmarkRunner.Run(infos)); //no args overload

    static void Catch(Action action)
    {
        Console.Write(benchmarkNumber++ + ":");
        try
        {
            action();
        }
        catch (Exception e)
        {
            Console.WriteLine(e.GetType().Name);
        }
    }
}

[DryJob]
public class Benchmark1
{
    [Benchmark] public void M1() { }
}

[DryJob]
public class Benchmark2
{
    [Benchmark] public void M1() { }
}

[DryJob]
public sealed class SealedBenchmarkClass
{
    [Benchmark]
    public void M1() { }
}

The Run* methods catch exceptions and write numbers (to make it easier to find an empty output)

ARGS=empty means args=Array.Empty<string>()

The output is the same for PR's args=empty and args=null (at least for these test cases).

nulls:

RunType(null as Type);
RunTypes(null as Type[]);
RunAssembly(null as Assembly);
RunInfo(null as BenchmarkRunInfo);
RunInfos(null as BenchmarkRunInfo[]);
RunTypeMethods(null as Type, null as MethodInfo[]);

Master args=null:

0:NullReferenceException
1:ArgumentNullException
2:NullReferenceException
3:NullReferenceException
4:ArgumentNullException
5:NullReferenceException

Master args=empty

0:ArgumentNullException
1:ArgumentNullException
2:NullReferenceException
3:NullReferenceException
4:ArgumentNullException
5:NullReferenceException

PR

0:ArgumentNullException
1:ArgumentNullException
2:ArgumentNullException
3:ArgumentNullException
4:ArgumentNullException
5:ArgumentNullException

RunT == BenchmarkRunner.Run<T>();

RunT<string>(DefaultConfig.Instance);
RunT<string>();
RunT<SealedBenchmarkClass>();

Master args=null:

RunT
6:
7:
8://Execute benchmark for SealedBenchmarkClass. Error in logs: "C:\src\github\BenchmarkDotNet-analyzers\src\ConsoleApp2\bin\Debug\net6.0\8982471e-ffa4-40ea-a316-bbc18fade75d\8982471e-ffa4-40ea-a316-bbc18fade75d.notcs(125,38): error CS0509: 'Runnable_0': cannot derive from sealed type 'SealedBenchmarkClass' [C:\src\github\BenchmarkDotNet-analyzers\src\ConsoleApp2\bin\Debug\net6.0\8982471e-ffa4-40ea-a316-bbc18fade75d\BenchmarkDotNet.Autogenerated.csproj]Build FAILED."```

Master args=empty

6:Type System.String is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
InvalidOperationException
7:Type System.String is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
InvalidOperationException
8:Type SealedBenchmarkClass is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
InvalidOperationException

PR

6:Type System.String is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
7:Type System.String is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
8:Type SealedBenchmarkClass is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.

RunTypes == BenchmarkRunner.Run(Type[]);

RunTypes(Array.Empty<Type>());
RunTypes(new Type[] { typeof(string) });
RunTypes(new Type[] { typeof(string), typeof(Benchmark1), typeof(object), typeof(List<string>) });
RunTypes(new Type[] { typeof(string), null });
RunTypes(new Type[] { null, typeof(Benchmark1) });

Master args=null:

9:
10:
11://Execute benchmark for Benchmark1.M1()
12:NullReferenceException
13:NullReferenceException

Master args=empty

9:No benchmarks to choose from. Make sure you provided public non-sealed non-static types with public [Benchmark] methods.10:Type System.String is invalid. Only public, non-generic (closed generic types with public parameterlessctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
11:Type System.String is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
12:Type System.String is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
13:ArgumentNullException

PR

9:No types provided.
10:Invalid Types:
  System.String
Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
11:Invalid Types:
  System.String
  System.Object
  System.Collections.Generic.List`1[[System.String, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]
Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
12:Null not allowed.
13:Null not allowed.

RunTypeMethods == BenchmarkRunner.Run(Type, MethodInfo[]);

MethodInfo strMethod = typeof(string).GetMethods().First(m => m.Name == "Remove");
MethodInfo benchmarkMethod = typeof(Benchmark1).GetMethod("M1");

RunTypeMethods(typeof(Benchmark1), new MethodInfo[] { benchmarkMethod, null });
RunTypeMethods(typeof(Benchmark1), new MethodInfo[] { strMethod });
RunTypeMethods(typeof(string), new MethodInfo[] { benchmarkMethod });

RunTypeMethods(typeof(string), new MethodInfo[] { });
RunTypeMethods(typeof(Benchmark1), new MethodInfo[] { });

RunTypeMethods(typeof(Benchmark1), new MethodInfo[] { benchmarkMethod, typeof(Benchmark2).GetMethod("M1") });
RunTypeMethods(typeof(Benchmark1), typeof(Benchmark2).GetMethods());

Master (no args overload):

14://Execute benchmark for Benchmark1.M1()
15:
16://Execute benchmark for string.M1(). Error in logs: "Generate Exception: Access to the path 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.7\10383f57-d463-4ae7-a453-27adb7bd915e\bin\Release\net6.0' is denied"```
17:
18:
19://Execute benchmark for Benchmark1.M1 twice
20://Execute benchmark for Benchmark1.M1

PR

14:Null not allowed.
15:Invalid methods:
  System.String.Remove
Methods must be of Benchmark1 type. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
16:Type System.String is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
17:Type System.String is invalid. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
18:No methods provided for Benchmark1.
19:Invalid methods:
  Benchmark2.M1
Methods must be of Benchmark1 type. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.
20:Invalid methods:
  Benchmark2.M1
  Benchmark2.GetType
  Benchmark2.ToString
  Benchmark2.Equals
  Benchmark2.GetHashCode
Methods must be of Benchmark1 type. Only public, non-generic (closed generic types with public parameterless ctors are supported), non-abstract, non-sealed, non-static types with public instance [Benchmark] method(s) are supported.

RunAssembly == BenchmarkRunner.Run(Assembly);

RunAssembly(typeof(string).Assembly);

Master args=null:

21:

Master args=empty

21:No benchmarks to choose from. Make sure you provided public non-sealed non-static types with public [Benchmark] methods.

PR

21:No benchmarks to choose from. Make sure you provided public non-sealed non-static types with public [Benchmark] methods.

RunInfo == BenchmarkRunner.Run(BenchmarkRunInfo);

 var config = DefaultConfig.Instance.CreateImmutableConfig();

RunInfo(new BenchmarkRunInfo(null, null, null));
RunInfo(new BenchmarkRunInfo(null, null, config));

RunInfo(new BenchmarkRunInfo(null, typeof(Benchmark1), null));
RunInfo(new BenchmarkRunInfo(null, typeof(Benchmark1), config));

RunInfo(new BenchmarkRunInfo(Array.Empty<BenchmarkCase>(), null, null));
RunInfo(new BenchmarkRunInfo(Array.Empty<BenchmarkCase>(), null, config));

RunInfo(new BenchmarkRunInfo(Array.Empty<BenchmarkCase>(), typeof(Benchmark1), null));
RunInfo(new BenchmarkRunInfo(Array.Empty<BenchmarkCase>(), typeof(Benchmark1), config));

Master (no args overload):

22:ArgumentNullException
23:ArgumentNullException
24:ArgumentNullException
25:ArgumentNullException
26:NullReferenceException
27:
28:NullReferenceException
29:

PR

22:BenchmarkRunInfo do not support null values.
23:BenchmarkRunInfo do not support null values.
24:BenchmarkRunInfo do not support null values.
25:BenchmarkRunInfo do not support null values.
26:BenchmarkRunInfo do not support null values.
27:BenchmarkRunInfo do not support null values.
28:BenchmarkRunInfo do not support null values.
29:BenchmarkRunInfo do not support null values.

RunInfos == BenchmarkRunner.Run(BenchmarkRunInfo[]);

To construct BenchmarkRunInfo we should call BenchmarkCase.Create (because ctor is internal), which is checking arguments itself. So I only checked arguments for null.

Aug 25 '22 21:08 YegorStepanov

A real PR output (without test numbers):

I like multiline with 2 spaces for invalid types/methods.

Aug 25 '22 21:08 YegorStepanov

There is a breaking change here: (Adam wrote half a year ago that this is expected)

var methods = typeof(MyClass).GetMethods(); // returns M() GetType() ToString() Equals() GetHashCode()
BenchmarkRunner.Run(typeof(MyClass));

public class MyClass{ [Benchmark] public void M() { } }

// Current: execute only correct benchmark methods: M()
// PR: display an error that GetType(), ToString(), Equals(), GetHashCode() are not correct benchmark methods

Oct 21 '22 14:10 YegorStepanov